E11.20 - Test Method Evaluation and Quality Control

ABSTRACT
The purpose of this guide is to provide guidelines for identifying the elements that comprise the test result of a test method and to illustrate how these elements combine into the test result. It covers the types of measurement scales used for expressing observations and test results. This guide provides information on the construction of test results from more elemental measurements.
SIGNIFICANCE AND USE
4.1 All test methods have an output in the form of a test result. This guide provides information on the construction of test results from more elemental measurements.  
4.2 A well-defined test result is necessary before any precision statements can be made about the test method.  
4.2.1 Form and Style for ASTM Standards,2 Section A21, requires that every test method shall contain a statement regarding its precision, preferably as a result of an interlaboratory test program. Reporting of such studies is described in Practice E177, which illustrates the development of test results from observations and test determinations.  
4.2.2 Precision statements for ASTM test methods are applicable to test results. They are not applicable to test determinations or observations, unless specifically and clearly indicated otherwise.
SCOPE
1.1 This standard provides guidelines for identifying the elements that comprise the test result of a test method and to illustrate how these elements combine into the test result.  
1.2 Types of measurement scales used for expressing observations and test results are discussed.  
1.3 No system of units is specified in this standard.  
1.4 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
This guide identifies statistical procedures for use in developing new test methods or revising or evaluating existing test methods, or both. It also cites statistical procedures especially useful in the application of test methods. This standard recommends what approaches may be taken and indicates which standards may be used to perform such assessments.
SIGNIFICANCE AND USE
4.1 The creation of a standardized test method generally follows a series of steps from inception to approval and ongoing use. In all such stages there are questions of how well the test method performs.  
4.1.1 Assessments of a new or existing test method generally involve statistical planning and analysis. This standard recommends what approaches may be taken and indicates which standards may be used to perform such assessments.  
4.2 This standard introduces a series of phases which are recommended to be considered during the life cycle of a test method as depicted in Fig. 1. These begin with a design phase where the standard is initially prepared. A development phase involves a variety of experiments that allow further refinement and understanding of how the test method performs within a laboratory. In an evaluation phase the test method is then examined by way of interlaboratory studies resulting in precision and bias statistics which are published in the standard. Finally, the test method is subject to a monitoring phase.
FIG. 1 Sequence of Steps  
4.3 All ASTM test methods are required to include statements on precision and bias.3  
4.4 Since ASTM began to require all test methods to have precision and bias statements that are based on interlaboratory studies, there has been increased concern regarding what statistical experiments and procedures to use during the development of the test methods. Although there exists a wide range of statistical procedures, there is a small group of generally accepted techniques that are beneficial to follow. This guide is designed to provide a brief overview of these procedures and to suggest an appropriate sequence of conducting these procedures.  
4.5 Statistical procedures often result in interpretations that are not absolutes. Sometimes the information obtained may be inadequate or incomplete, which may lead to additional questions and the need for further experimentation. Information outside the data is also impo...
SCOPE
1.1 This guide identifies statistical procedures for use in developing new test methods or revising or evaluating existing test methods, or both.  
1.2 This guide also cites statistical procedures especially useful in the application of test methods.  
1.3 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
This practice provides a general methodology for evaluating single-stage or multiple-stage acceptance procedures which involve a quality characteristic measured on a numerical scale. This methodology computes, at a prescribed confidence level, a lower bound on the probability of passing an acceptance procedure, using estimates of the parameters of the distribution of test results from a sampled population.
SIGNIFICANCE AND USE
4.1 This practice considers inspection procedures that may involve multiple-stage sampling, where at each stage one can decide to accept or to continue sampling, and the decision to reject is deferred until the last stage.  
4.1.1 At each stage there are one or more acceptance criteria on the test results; for example, limits on each individual test result, or limits on statistics based on the sample of test results, such as the average, standard deviation, or coefficient of variation (relative standard deviation).  
4.2 The methodology in this practice defines an acceptance region for a set of test results from the sampled population such that, at a prescribed confidence level, the probability that a sample from the population will pass the acceptance procedure is greater than or equal to a prespecified lower bound.  
4.2.1 Having test results fall in the acceptance region is not equivalent to passing the acceptance procedure, but provides assurance that a sample would pass the acceptance procedure with a specified probability.  
4.2.2 This information can be used for process demonstration, validation of test methods, and qualification of instruments, processes, and materials.  
4.2.3 This information can be used for lot release (acceptance), but the lower bound may be conservative in some cases.  
4.2.4 If the results are to be applied to future test results from the same process, then it is assumed that the process is stable and predictable. If this is not the case then there can be no guarantee that the probability estimates would be valid predictions of future process performance.  
4.3 This methodology was originally developed (1-4)3 for use in two specific quality characteristics of drug products in the pharmaceutical industry but will be applicable for acceptance procedures in all industries.  
4.4 Mathematical derivations would be required that are specific to the individual criteria of each test.
SCOPE
1.1 This practice provides a general methodology for evaluating single-stage or multiple-stage acceptance procedures which involve a quality characteristic measured on a numerical scale. This methodology computes, at a prescribed confidence level, a lower bound on the probability of passing an acceptance procedure, using estimates of the parameters of the distribution of test results from a sampled population.  
1.2 For a prescribed lower probability bound, the methodology can also generate an acceptance limit table, which defines a set of test method outcomes (for example, sample averages and standard deviations) that would pass the acceptance procedure at a prescribed confidence level.  
1.3 This approach may be used for demonstrating compliance with in-process, validation, or lot-release specifications.  
1.4 The system of units for this practice is not specified.  
1.5 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.6 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method. This practice is also concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements. ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility and knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values or between data sources.
SIGNIFICANCE AND USE
4.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177. Knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values (such as specification limits) or between data sources (different laboratories, instruments, etc.).  
4.1.1 When a test method is applied to a large number of portions of a material that are as nearly alike as possible, the test results obtained will not all have the same value. A measure of the degree of agreement among these test results describes the precision of the test method for that material. Numerical measures of the variability between such test results provide inverse measures of the precision of the test method. Greater variability implies smaller (that is, poorer) precision and larger imprecision.  
4.1.2 Precision is reported as a standard deviation, coefficient of variation (relative standard deviation), variance, or a precision limit (a data range indicating no statistically significant difference between test results).  
4.1.3 This practice is designed only to estimate the precision of a test method. However, when accepted reference values are available for the property levels, the test result data obtained according to this practice may be used in estimating the bias of the test method. For a discussion of bias estimation and the relationships between precision, bias, and accuracy, see Practice E177.  
4.2 The procedures presented in this practice consist of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data.  
4.2.1 The planning phase includes forming the ILS task group, the study design, selection, and number of participating laborato...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements.  
1.4.1 This practice ...

ABSTRACT
This practice describes methods for the statistical analysis of laboratory results obtained from interlaboratory proficiency testing programs. As in accordance with Practice E1301, proficiency testing is the use of inter-laboratory comparisons for the determination of laboratory testing or measurement performance. The methods provide direction for assessing and categorizing the performance of individual laboratories based on the relative likelihood of occurrence of their test results, and for determining estimates of testing variation associated with repeatability and reproducibility. Assumptions are that a majority of the participating laboratories execute the test method properly and that samples are of sufficient homogeneity that the testing results represent results obtained from each laboratory testing essentially the same material. Each laboratory receives the same instructions or protocol.
SIGNIFICANCE AND USE
5.1 This practice is specifically designed to describe simple robust statistical methods for use in proficiency testing programs.  
5.2 Proficiency testing programs can use the methods in this practice for the purpose of comparing testing results obtained from a group of participating laboratories. The practice describes evaluation of individual laboratory results using the interquartile range and Tukey inner and outer fences.  
5.3 In addition, the data obtained in proficiency testing programs may contain information regarding repeatability (within-lab) and reproducibility (between-lab) testing variation. Repeatability information is possible only if the program uses more than one sample. See Method B. Proficiency testing programs often have a greater number of participants than might be available for conducting an interlaboratory study to determine the precision of a test method (such as described in Practice E691). Precision estimates obtained for the larger number of participants in a proficiency testing program, along with the corresponding wider variation of test conditions, can provide useful information to standards developers regarding the precision of test results that can be expected for a test method when in actual use in the general testing community.  
5.4 To estimate the precision of a test method, the participants must use the same test method to obtain their test results, and testing must be performed under the conditions required for repeatability and reproducibility. The precision estimates are applicable to the property levels and material types included in the testing program. The precision of a test method may vary considerably for different material types and at different property levels.  
5.5 This practice may be useful to proficiency testing program administrators and provides examples of statistical methods along with explanations of some of the advantages of the suggested methods of analysis. The analyses resulting from the applicatio...
SCOPE
1.1 This practice describes methods for the statistical analysis of laboratory results obtained from interlaboratory proficiency testing programs. As in accordance with Practice E1301, proficiency testing is the use of interlaboratory comparisons for the determination of laboratory testing or measurement performance. Conversely, collaborative study (or collaborative trial) is the use of interlaboratory comparisons for the determination of the precision of a test method, as covered by Practice E691.  
1.1.1 Method A covers testing programs using single test results obtained by testing a single sample (each laboratory submits a single test result).  
1.1.2 Method B covers testing programs using paired test results obtained by testing two samples (each laboratory submits one test result for each of the two samples). The two samples should be of the same material or two materials similar enough to have approximately the same degree of variation in test results.  
1.2 Methods A and B are applicable to proficiency test...

ABSTRACT
This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources to determine if their true means or variances differ by no more than predetermined limits. This standard provides guidance on experiments and statistical methods needed to demonstrate that the test results from a modified testing process are equivalent to those from the current testing process, where equivalence is defined as agreement within a prescribed limit, termed an equivalence limit.
SIGNIFICANCE AND USE
4.1 Laboratories conducting routine testing have a continuing need to make improvements in their testing processes. In these situations it must be demonstrated that any changes will neither cause an undesirable shift in the test results from the current testing process nor substantially affect a performance characteristic of the test method. This standard provides guidance on experiments and statistical methods needed to demonstrate that the test results from a modified testing process are equivalent to those from the current testing process, where equivalence is defined as agreement within a prescribed limit, termed an equivalence limit.  
4.1.1 The equivalence limit, which represents a worst-case difference or ratio, is determined prior to the equivalence test and its value is usually set by consensus among subject-matter experts.  
4.1.2 Examples of modifications to the testing process include, but are not limited, to the following:  
(1) Changes to operating levels in the steps of the test method procedure,
(2) Installation of new instruments, apparatus, or sources of reagents and test materials,
(3) Evaluation of new personnel performing the testing, and
(4) Transfer of testing to a new location.  
4.1.3 Examples of performance characteristics directly applicable to the test method include bias, precision, sensitivity, specificity, linearity, and range. Additional characteristics are test cost and elapsed time needed to conduct the test procedure.  
4.2 Equivalence studies are performed by a designed experiment that generates test results from the modified and current testing procedures on the same types of materials that are routinely tested. The design of the experiment depends on the type of equivalence needed as discussed below. Experiment design and execution for various objectives is discussed in Section 5.  
4.2.1 Means equivalence is concerned with a potential shift in the mean test result in either direction due to a modification in the te...
SCOPE
1.1 This practice provides statistical methodology for conducting equivalence studies on numerical data from two sources of test results to determine if their true means, variances, or other parameters differ by no more than predetermined limits.  
1.2 Applications include (1) equivalence studies for bias against an accepted reference value, (2) determining means equivalence of two test methods, test apparatus, instruments, reagent sources, or operators within a laboratory or equivalence of two laboratories in a method transfer, and (3) determining non-inferiority of a modified test procedure versus a current test procedure with respect to a performance characteristic.  
1.3 The guidance in this standard applies to experiments conducted either on a single material at a given level of the test result or on multiple materials covering a selected range of test results.  
1.4 Guidance is given for determining the amount of data required for an equivalence study. The control of risks associated with the equivalence decision is discussed.  
1.5 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine t...

SIGNIFICANCE AND USE
4.1 A ruggedness test is a special application of a statistically designed experiment that makes changes in the test method variables, called factors, and then calculates the subsequent effect of those changes upon the test results. Factors are features of the test method or of the laboratory environment that are known to vary across laboratories and are subject to control by the test method.  
4.1.1 Statistical design enables more efficient and cost-effective determination of the factor effects than would be achieved if separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed for evaluating quantitative test methods.  
4.2 In ruggedness testing, the two levels (settings) for each factor are chosen to use moderate separations between the high and low settings. In general, if there is an underlying difference between the levels, then the size of effects will increase with increased separation between the high and low settings of the factors. A run is an execution of the test method under prescribed settings of each of the factors under study. A ruggedness test consists of a set of runs.  
4.3 A ruggedness test is usually conducted within a single laboratory on uniform material, so that the effects of changing only the factors are measured. The results may then be used to assist in determining the degree of control required of factors described in the test method.  
4.4 Ruggedness testing should precede an interlaboratory (round robin) study to correct any deficiencies in the test method and may also be part of the validation phase of developing a standard test method as described in Guide E1488.  
4.5 This standard discusses design and analysis of ruggedness testing in Section 5 and contains an example of a basic eight run design. Some caution must be used in interpretation of results, since interaction effects may be present. These effects are present when a factor effect changes with the ...
SCOPE
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.  
1.2 This practice restricts itself to experimental designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.  
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.5 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

SIGNIFICANCE AND USE
5.1 This guide presents techniques and guidance for evaluating and assuring homogeneity of individual samples or bulk materials and can be used for either interlaboratory or intra-laboratory studies. The types of studies include, but are not limited to, studies to determine precision estimates for test methods, proficiency testing programs, and studies related to quality control of testing within a single laboratory.  
5.2 Because the test results of any laboratory study are affected by the quality of the samples tested, producing homogeneous samples and determining the degree of homogeneity is important for interpreting the results of the study.  
5.3 Five techniques are presented in this guide to evaluate sample homogeneity for a range of circumstances and degrees of rigor. The circumstances under which the studies are conducted and the degree of rigor required may differ. The user should consider the circumstances listed in each technique to determine which is appropriate for the study at hand.  
5.4 Each of the Techniques 1, 2, and 3 provides a procedure for testing and evaluating sample homogeneity when replicate testing of the samples is possible. Technique 4 provides a plan to evaluate sample homogeneity when replicate testing is not possible. Technique 5 recommends practices for producing homogeneous samples for circumstances when homogeneity testing is not possible.  
5.5 When the conditions of adequate within-sample homogeneity and between-sample homogeneity are satisfied, any differences in test results on multiple samples can reasonably be attributed to testing variation and not due to sample variation.  
5.6 When differences within or between samples are discovered and the samples are deemed insufficiently homogeneous, the sample preparation process can be improved or corrected and a new set of samples can be prepared. Or, in cases where the sample homogeneity cannot be improved or for other reasons when the samples must be used, the method of eval...
SCOPE
1.1 This guide presents techniques and guidance for evaluating and assuring homogeneity of individual samples and bulk materials used for interlaboratory and intra-laboratory studies.  
1.2 This guide is applicable to samples and reference materials used for proficiency testing programs and for interlaboratory studies to determine precision estimates for test methods. It may also be useful for activities related to quality control of testing within a single laboratory.  
1.3 Five techniques are presented for assessing sample homogeneity. The five techniques are not an exhaustive list of available techniques for assessing homogeneity of samples, but the techniques were chosen to cover a range of circumstances (and various degrees of rigor required) for laboratory studies of various types and purposes.  
1.4 Each of the first four techniques provides a scheme for testing for homogeneity and a statistical procedure for evaluating the results of the homogeneity testing. The circumstances are described for which each of the techniques is suited.  
1.5 For circumstances when homogeneity testing is not possible, the fifth technique provides guidance for producing homogeneous samples.  
1.6 The appendixes of this guide provide example spreadsheets for Techniques 1, 2, 3, and 4.  
1.7 This guide is not intended for evaluation of certified reference materials (CRMs) or materials used for calibration.  
1.8 Units—The system of units for this standard is not specified. Dimensional quantities in the standard are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.9 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to ...

ABSTRACT
The purpose of this practice is to present concepts necessary to the understanding of the terms “precision” and “bias” as used in quantitative test methods. This practice also describes methods of expressing precision and bias and, in a final section, gives examples of how statements on precision and bias may be written for ASTM test methods. A statement of precision allows potential users of a test method to assess in general terms the test method’s usefulness with respect to variability in proposed applications.
SIGNIFICANCE AND USE
4.1 Part A of the “Blue Book,” Form and Style for ASTM Standards, requires that all test methods include statements of precision and bias. This practice discusses these two concepts and provides guidance for their use in statements about test methods.  
4.2 Precision—A statement of precision allows potential users of a test method to assess in general terms the test method’s usefulness with respect to variability in proposed applications. A statement of precision is not intended to exhibit values that can be exactly duplicated in every user’s laboratory. Instead, the statement provides guidelines as to the magnitude of variability that can be expected between test results when the method is used in one, or in two or more, reasonably competent laboratories. For a discussion of precision, see 8.1.  
4.3 Bias—A statement of bias furnishes guidelines on the relationship between a set of typical test results produced by the test method under specific test conditions and a related set of accepted reference values (see 9.1).  
4.3.1 An alternative term for bias is trueness, which has a positive connotation, in that greater bias is associated with less favorable trueness. Trueness is the systematic component of accuracy.  
4.4 Accuracy—The term “accuracy,” used in earlier editions of Practice E177, embraces both precision and bias (see 9.3).
SCOPE
1.1 The purpose of this practice is to present concepts necessary to the understanding of the terms “precision” and “bias” as used in quantitative test methods. This practice also describes methods of expressing precision and bias and, in a final section, gives examples of how statements on precision and bias may be written for ASTM test methods.  
1.2 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.3 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
This guide covers key elements of an evaluation of a laboratory’s measurement practices and the statistical analysis of the resulting data, as well as addresses an evaluation that covers a broad range of in-house quality measurements, some of which may be directly related to accreditation requirements. It provides an overview of the documentation needed for verification and monitoring of the practices used in the laboratory for measurement. In addition, it guides the user in verifying that the extent of documentation and the quality of statistical evaluations performed on the data being generated is sufficient. This guide is also intended to provide guidance for laboratory quality managers, accrediting bodies and assessors in evaluating the measurement practices of a laboratory and statistically analyzing the resulting data from these practices.
SIGNIFICANCE AND USE
4.1 This guide is intended to provide guidance for laboratory quality managers, accrediting bodies and assessors in evaluating the measurement practices of a laboratory, the protocol for statistically analyzing the resulting data from these practices, and the statistical results from these practices.  
4.2 This guide is generic in the sense that it covers the entire range of in-house quality measurement practices found in a testing laboratory, and the results of the described evaluation may be used by accrediting agencies for assessment purposes to determine whether their requirements can be satisfied through the laboratory’s existing quality data program.  
4.3 It is not the intent of this guide to serve as sole criterion for evaluating and accrediting laboratories. Evaluation of measurement practices is only one aspect in a comprehensive quality program.
SCOPE
1.1 This guide covers key elements of an evaluation of a laboratory’s measurement practices and the statistical analysis of the resulting data. This guide addresses an evaluation that covers a broad range of in-house quality measurements, some of which may be directly related to accreditation requirements.  
1.2 This guide provides an overview of the documentation needed for verification and monitoring of the practices used in the laboratory for measurement. In addition, it guides the user in verifying that the extent of documentation and the quality of statistical evaluations performed on the data being generated is sufficient. The user is advised to fully document all work covered by the scope of this guide as a general principle of laboratory practice and for audit purposes, whether internal or external.  
1.3 This guide is not designed to be exhaustive for all aspects of work realized under its scope. The user is encouraged to thoroughly realize (achieve in practice) the principles set forth in this guide, consulting other relevant standards and industry documents when appropriate.  
1.4 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
This practice provides a general methodology for evaluating single-stage or multiple-stage acceptance procedures which involve a quality characteristic measured on a numerical scale. This methodology computes, at a prescribed confidence level, a lower bound on the probability of passing an acceptance procedure, using estimates of the parameters of the distribution of test results from a sampled population.
SIGNIFICANCE AND USE
4.1 This practice considers inspection procedures that may involve multiple-stage sampling, where at each stage one can decide to accept or to continue sampling, and the decision to reject is deferred until the last stage.  
4.1.1 At each stage there are one or more acceptance criteria on the test results; for example, limits on each individual test result, or limits on statistics based on the sample of test results, such as the average, standard deviation, or coefficient of variation (relative standard deviation).  
4.2 The methodology in this practice defines an acceptance region for a set of test results from the sampled population such that, at a prescribed confidence level, the probability that a sample from the population will pass the acceptance procedure is greater than or equal to a prespecified lower bound.  
4.2.1 Having test results fall in the acceptance region is not equivalent to passing the acceptance procedure, but provides assurance that a sample would pass the acceptance procedure with a specified probability.  
4.2.2 This information can be used for process demonstration, validation of test methods, and qualification of instruments, processes, and materials.  
4.2.3 This information can be used for lot release (acceptance), but the lower bound may be conservative in some cases.  
4.2.4 If the results are to be applied to future test results from the same process, then it is assumed that the process is stable and predictable. If this is not the case then there can be no guarantee that the probability estimates would be valid predictions of future process performance.  
4.3 This methodology was originally developed (1-4)3 for use in two specific quality characteristics of drug products in the pharmaceutical industry but will be applicable for acceptance procedures in all industries.  
4.4 Mathematical derivations would be required that are specific to the individual criteria of each test.
SCOPE
1.1 This practice provides a general methodology for evaluating single-stage or multiple-stage acceptance procedures which involve a quality characteristic measured on a numerical scale. This methodology computes, at a prescribed confidence level, a lower bound on the probability of passing an acceptance procedure, using estimates of the parameters of the distribution of test results from a sampled population.  
1.2 For a prescribed lower probability bound, the methodology can also generate an acceptance limit table, which defines a set of test method outcomes (for example, sample averages and standard deviations) that would pass the acceptance procedure at a prescribed confidence level.  
1.3 This approach may be used for demonstrating compliance with in-process, validation, or lot-release specifications.  
1.4 The system of units for this practice is not specified.  
1.5 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.6 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
The purpose of this guide is to provide guidelines for identifying the elements that comprise the test result of a test method and to illustrate how these elements combine into the test result. It covers the types of measurement scales used for expressing observations and test results. This guide provides information on the construction of test results from more elemental measurements.
SIGNIFICANCE AND USE
4.1 All test methods have an output in the form of a test result. This guide provides information on the construction of test results from more elemental measurements.  
4.2 A well defined test result is necessary before any precision statements can be made about the test method.  
4.2.1 Form and Style for ASTM Standards,2 Section A21, requires that every test method shall contain a statement regarding its precision, preferably as a result of an interlaboratory test program. Reporting of such studies is described in Practice E177, which illustrates the development of test results from observations and test determinations.  
4.2.2 Precision statements for ASTM test methods are applicable to test results. They are not applicable to test determinations or observations, unless specifically and clearly indicated otherwise.
SCOPE
1.1 The purpose of this guide is to provide guidelines for identifying the elements that comprise the test result of a test method and to illustrate how these elements combine into the test result.  
1.2 Types of measurement scales used for expressing observations and test results are discussed.  
1.3 No system of units is specified in this standard.  
1.4 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
This guide identifies statistical procedures for use in developing new test methods or revising or evaluating existing test methods, or both. It also cites statistical procedures especially useful in the application of test methods. This standard recommends what approaches may be taken and indicates which standards may be used to perform such assessments.
SIGNIFICANCE AND USE
4.1 The creation of a standardized test method generally follows a series of steps from inception to approval and ongoing use. In all such stages there are questions of how well the test method performs.  
4.1.1 Assessments of a new or existing test method generally involve statistical planning and analysis. This standard recommends what approaches may be taken and indicates which standards may be used to perform such assessments.  
4.2 This standard introduces a series of phases which are recommended to be considered during the life cycle of a test method as depicted in Fig. 1. These begin with a design phase where the standard is initially prepared. A development phase involves a variety of experiments that allow further refinement and understanding of how the test method performs within a laboratory. In an evaluation phase the test method is then examined by way of interlaboratory studies resulting in precision and bias statistics which are published in the standard. Finally, the test method is subject to a monitoring phase.
FIG. 1 Sequence of Steps  
4.3 All ASTM test methods are required to include statements on precision and bias.3  
4.4 Since ASTM began to require all test methods to have precision and bias statements that are based on interlaboratory test methods, there has been increased concern regarding what statistical experiments and procedures to use during the development of the test methods. Although there exists a wide range of statistical procedures, there is a small group of generally accepted techniques that are beneficial to follow. This guide is designed to provide a brief overview of these procedures and to suggest an appropriate sequence of carrying out these procedures.  
4.5 Statistical procedures often result in interpretations that are not absolutes. Sometimes the information obtained may be inadequate or incomplete, which may lead to additional questions and the need for further experimentation. Information outside the data is al...
SCOPE
1.1 This guide identifies statistical procedures for use in developing new test methods or revising or evaluating existing test methods, or both.  
1.2 This guide also cites statistical procedures especially useful in the application of test methods.  
1.3 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method. This practice is also concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements. ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility and knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values or between data sources.
SIGNIFICANCE AND USE
4.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177. Knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values (such as specification limits) or between data sources (different laboratories, instruments, etc.).  
4.1.1 When a test method is applied to a large number of portions of a material that are as nearly alike as possible, the test results obtained will not all have the same value. A measure of the degree of agreement among these test results describes the precision of the test method for that material. Numerical measures of the variability between such test results provide inverse measures of the precision of the test method. Greater variability implies smaller (that is, poorer) precision and larger imprecision.  
4.1.2 Precision is reported as a standard deviation, coefficient of variation (relative standard deviation), variance, or a precision limit (a data range indicating no statistically significant difference between test results).  
4.1.3 This practice is designed only to estimate the precision of a test method. However, when accepted reference values are available for the property levels, the test result data obtained according to this practice may be used in estimating the bias of the test method. For a discussion of bias estimation and the relationships between precision, bias, and accuracy, see Practice E177.  
4.2 The procedures presented in this practice consist of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data.  
4.2.1 The planning phase includes forming the ILS task group, the study design, selection, and number of participating laborato...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements.  
1.4.1 This practice ...

ABSTRACT
This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method. This practice is also concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements. ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility and knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values or between data sources.
SIGNIFICANCE AND USE
4.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177. Knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values (such as specification limits) or between data sources (different laboratories, instruments, etc.).  
4.1.1 When a test method is applied to a large number of portions of a material that are as nearly alike as possible, the test results obtained will not all have the same value. A measure of the degree of agreement among these test results describes the precision of the test method for that material. Numerical measures of the variability between such test results provide inverse measures of the precision of the test method. Greater variability implies smaller (that is, poorer) precision and larger imprecision.  
4.1.2 Precision is reported as a standard deviation, coefficient of variation (relative standard deviation), variance, or a precision limit (a data range indicating no statistically significant difference between test results).  
4.1.3 This practice is designed only to estimate the precision of a test method. However, when accepted reference values are available for the property levels, the test result data obtained according to this practice may be used in estimating the bias of the test method. For a discussion of bias estimation and the relationships between precision, bias, and accuracy, see Practice E177.  
4.2 The procedures presented in this practice consist of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data.  
4.2.1 The planning phase includes forming the ILS task group, the study design, selection, and number of participating laborato...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements.  
1.4.1 This practice ...

ABSTRACT
This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method. This practice is also concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements. ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility and knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values or between data sources.
SIGNIFICANCE AND USE
4.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177. Knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values (such as specification limits) or between data sources (different laboratories, instruments, etc.).  
4.1.1 When a test method is applied to a large number of portions of a material that are as nearly alike as possible, the test results obtained will not all have the same value. A measure of the degree of agreement among these test results describes the precision of the test method for that material. Numerical measures of the variability between such test results provide inverse measures of the precision of the test method. Greater variability implies smaller (that is, poorer) precision and larger imprecision.  
4.1.2 Precision is reported as a standard deviation, coefficient of variation (relative standard deviation), variance, or a precision limit (a data range indicating no statistically significant difference between test results).  
4.1.3 This practice is designed only to estimate the precision of a test method. However, when accepted reference values are available for the property levels, the test result data obtained according to this practice may be used in estimating the bias of the test method. For a discussion of bias estimation and the relationships between precision, bias, and accuracy, see Practice E177.  
4.2 The procedures presented in this practice consist of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data.  
4.2.1 The planning phase includes forming the ILS task group, the study design, selection, and number of participating laborato...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements.  
1.4.1 This practice ...

ABSTRACT
This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources to determine if their true means or variances differ by no more than predetermined limits. This standard provides guidance on experiments and statistical methods needed to demonstrate that the test results from a modified testing process are equivalent to those from the current testing process, where equivalence is defined as agreement within a prescribed limit, termed an equivalence limit.
SIGNIFICANCE AND USE
4.1 Laboratories conducting routine testing have a continuing need to make improvements in their testing processes. In these situations it must be demonstrated that any changes will neither cause an undesirable shift in the test results from the current testing process nor substantially affect a performance characteristic of the test method. This standard provides guidance on experiments and statistical methods needed to demonstrate that the test results from a modified testing process are equivalent to those from the current testing process, where equivalence is defined as agreement within a prescribed limit, termed an equivalence limit.  
4.1.1 The equivalence limit, which represents a worst-case difference or ratio, is determined prior to the equivalence test and its value is usually set by consensus among subject-matter experts.  
4.1.2 Examples of modifications to the testing process include, but are not limited, to the following:  
(1) Changes to operating levels in the steps of the test method procedure,
(2) Installation of new instruments, apparatus, or sources of reagents and test materials,
(3) Evaluation of new personnel performing the testing, and
(4) Transfer of testing to a new location.  
4.1.3 Examples of performance characteristics directly applicable to the test method include bias, precision, sensitivity, specificity, linearity, and range. Additional characteristics are test cost and elapsed time needed to conduct the test procedure.  
4.2 Equivalence testing is performed by a designed experiment that generates test results from the modified and current testing procedures on the same types of materials that are routinely tested. The design of the experiment depends on the type of equivalence needed as discussed below. Experiment design and execution for various objectives is discussed in Section 5.  
4.2.1 Means equivalence is concerned with a potential shift in the mean test result in either direction due to a modification in the tes...
SCOPE
1.1 This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources of test results to determine if their true means, variances, or other parameters differ by no more than predetermined limits.  
1.2 Applications include (1) equivalence testing for bias against an accepted reference value, (2) determining means equivalence of two test methods, test apparatus, instruments, reagent sources, or operators within a laboratory or equivalence of two laboratories in a method transfer, and (3) determining non-inferiority of a modified test procedure versus a current test procedure with respect to a performance characteristic.  
1.3 The guidance in this standard applies to experiments conducted either on a single material at a given level of the test result or on multiple materials covering a selected range of test results.  
1.4 Guidance is given for determining the amount of data required for an equivalence trial. The control of risks associated with the equivalence decision is discussed.  
1.5 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine t...

ABSTRACT
This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources to determine if their true means or variances differ by no more than predetermined limits. This standard provides guidance on experiments and statistical methods needed to demonstrate that the test results from a modified testing process are equivalent to those from the current testing process, where equivalence is defined as agreement within a prescribed limit, termed an equivalence limit.
SIGNIFICANCE AND USE
4.1 Laboratories conducting routine testing have a continuing need to make improvements in their testing processes. In these situations it must be demonstrated that any changes will neither cause an undesirable shift in the test results from the current testing process nor substantially affect a performance characteristic of the test method. This standard provides guidance on experiments and statistical methods needed to demonstrate that the test results from a modified testing process are equivalent to those from the current testing process, where equivalence is defined as agreement within a prescribed limit, termed an equivalence limit.  
4.1.1 The equivalence limit, which represents a worst-case difference or ratio, is determined prior to the equivalence test and its value is usually set by consensus among subject-matter experts.  
4.1.2 Examples of modifications to the testing process include, but are not limited, to the following:  
(1) Changes to operating levels in the steps of the test method procedure,
(2) Installation of new instruments, apparatus, or sources of reagents and test materials,
(3) Evaluation of new personnel performing the testing, and
(4) Transfer of testing to a new location.  
4.1.3 Examples of performance characteristics directly applicable to the test method include bias, precision, sensitivity, specificity, linearity, and range. Additional characteristics are test cost and elapsed time needed to conduct the test procedure.  
4.2 Equivalence testing is performed by a designed experiment that generates test results from the modified and current testing procedures on the same types of materials that are routinely tested. The design of the experiment depends on the type of equivalence needed as discussed below. Experiment design and execution for various objectives is discussed in Section 5.  
4.2.1 Means equivalence is concerned with a potential shift in the mean test result in either direction due to a modification in the tes...
SCOPE
1.1 This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources of test results to determine if their true means, variances, or other parameters differ by no more than predetermined limits.  
1.2 Applications include (1) equivalence testing for bias against an accepted reference value, (2) determining means equivalence of two test methods, test apparatus, instruments, reagent sources, or operators within a laboratory or equivalence of two laboratories in a method transfer, and (3) determining non-inferiority of a modified test procedure versus a current test procedure with respect to a performance characteristic.  
1.3 The guidance in this standard applies to experiments conducted either on a single material at a given level of the test result or on multiple materials covering a selected range of test results.  
1.4 Guidance is given for determining the amount of data required for an equivalence trial. The control of risks associated with the equivalence decision is discussed.  
1.5 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine t...

SIGNIFICANCE AND USE
4.1 A ruggedness test is a special application of a statistically designed experiment that makes changes in the test method variables, called factors, and then calculates the subsequent effect of those changes upon the test results. Factors are features of the test method or of the laboratory environment that are known to vary across laboratories and are subject to control by the test method.  
4.1.1 Statistical design enables more efficient and cost-effective determination of the factor effects than would be achieved if separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed for evaluating quantitative test methods.  
4.2 In ruggedness testing, the two levels (settings) for each factor are chosen to use moderate separations between the high and low settings. In general, if there is an underlying difference between the levels, then the size of effects will increase with increased separation between the high and low settings of the factors. A run is an execution of the test method under prescribed settings of each of the factors under study. A ruggedness test consists of a set of runs.  
4.3 A ruggedness test is usually conducted within a single laboratory on uniform material, so that the effects of changing only the factors are measured. The results may then be used to assist in determining the degree of control required of factors described in the test method.  
4.4 Ruggedness testing should precede an interlaboratory (round robin) study to correct any deficiencies in the test method and may also be part of the validation phase of developing a standard test method as described in Guide E1488.  
4.5 This standard discusses design and analysis of ruggedness testing in Section 5 and contains an example of a basic eight run design. Some caution must be used in interpretation of results, since interaction effects may be present. These effects are present when a factor effect changes with the ...
SCOPE
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.  
1.2 This practice restricts itself to experimental designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.  
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.5 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
The purpose of this practice is to present concepts necessary to the understanding of the terms “precision” and “bias” as used in quantitative test methods. This practice also describes methods of expressing precision and bias and, in a final section, gives examples of how statements on precision and bias may be written for ASTM test methods. A statement of precision allows potential users of a test method to assess in general terms the test method’s usefulness with respect to variability in proposed applications.
In this practice only quantitative test methods that produce numerical results are considered. Also, the word “material” is used to mean material, product, system or service; the word “property” is used herein to mean that a quantitative test result can be obtained that describes a characteristic or a quality, or some other aspect of the material; and “test method” refers to both the document and the procedure described therein for obtaining a quantitative test result for one property.
SIGNIFICANCE AND USE
4.1 Part A of the “Blue Book,” Form and Style for ASTM Standards, requires that all test methods include statements of precision and bias. This practice discusses these two concepts and provides guidance for their use in statements about test methods.  
4.2 Precision—A statement of precision allows potential users of a test method to assess in general terms the test method’s usefulness with respect to variability in proposed applications. A statement of precision is not intended to exhibit values that can be exactly duplicated in every user’s laboratory. Instead, the statement provides guidelines as to the magnitude of variability that can be expected between test results when the method is used in one, or in two or more, reasonably competent laboratories. For a discussion of precision, see 8.1.  
4.3 Bias—A statement of bias furnishes guidelines on the relationship between a set of typical test results produced by the test method under specific test conditions and a related set of accepted reference values (see 9.1).  
4.3.1 An alternative term for bias is trueness, which has a positive connotation, in that greater bias is associated with less favorable trueness. Trueness is the systematic component of accuracy.  
4.4 Accuracy—The term “accuracy,” used in earlier editions of Practice E177, embraces both precision and bias (see 9.3).
SCOPE
1.1 The purpose of this practice is to present concepts necessary to the understanding of the terms “precision” and “bias” as used in quantitative test methods. This practice also describes methods of expressing precision and bias and, in a final section, gives examples of how statements on precision and bias may be written for ASTM test methods.  
1.2 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.3 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

SIGNIFICANCE AND USE
4.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177. Knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values (such as specification limits) or between data sources (different laboratories, instruments, etc.).  
4.1.1 When a test method is applied to a large number of portions of a material that are as nearly alike as possible, the test results obtained will not all have the same value. A measure of the degree of agreement among these test results describes the precision of the test method for that material. Numerical measures of the variability between such test results provide inverse measures of the precision of the test method. Greater variability implies smaller (that is, poorer) precision and larger imprecision.  
4.1.2 Precision is reported as a standard deviation, coefficient of variation (relative standard deviation), variance, or a precision limit (a data range indicating no statistically significant difference between test results).  
4.1.3 This practice is designed only to estimate the precision of a test method. However, when accepted reference values are available for the property levels, the test result data obtained according to this practice may be used in estimating the bias of the test method. For a discussion of bias estimation and the relationships between precision, bias, and accuracy, see Practice E177.  
4.2 The procedures presented in this practice consist of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data.  
4.2.1 The planning phase includes forming the ILS task group, the study design, selection and number of participating laborator...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements.  
1.4.1 This practice does not cover methods in which the measurement is a categorization; however, for many practical purposes categorical outcomes can be scored, such as zero-one scoring for binary measurements or as integers, ranks for example, for well-ordered categories and then the test result can be defined as an average, or other summary statistic, of several individual scores.  
1.5 This standard may involve hazardous materials, operations, and equipment. This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.6 This international standard was developed...

ABSTRACT
This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method. This practice is also concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements. ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility and knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values or between data sources.
The procedures presented in this practice consist of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data.
SIGNIFICANCE AND USE
4.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177. Knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values (such as specification limits) or between data sources (different laboratories, instruments, etc.).  
4.1.1 When a test method is applied to a large number of portions of a material that are as nearly alike as possible, the test results obtained will not all have the same value. A measure of the degree of agreement among these test results describes the precision of the test method for that material. Numerical measures of the variability between such test results provide inverse measures of the precision of the test method. Greater variability implies smaller (that is, poorer) precision and larger imprecision.  
4.1.2 Precision is reported as a standard deviation, coefficient of variation (relative standard deviation), variance, or a precision limit (a data range indicating no statistically significant difference between test results).  
4.1.3 This practice is designed only to estimate the precision of a test method. However, when accepted reference values are available for the property levels, the test result data obtained according to this practice may be used in estimating the bias of the test method. For a discussion of bias estimation and the relationships between precision, bias, and accuracy, see Practice E177.  
4.2 The procedures presented in this practice consist of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data.  
4.2.1 The planning phase includes forming the ILS task group, the study design, selection and number of participating laborator...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with ...

ABSTRACT
This practice provides a general methodology for evaluating single-stage or multiple-stage acceptance procedures which involve a quality characteristic measured on a numerical scale. This methodology computes, at a prescribed confidence level, a lower bound on the probability of passing an acceptance procedure, using estimates of the parameters of the distribution of test results from a sampled population.
This practice considers inspection procedures that may involve multiple-stage sampling, where at each stage one can decide to accept or to continue sampling, and the decision to reject is deferred until the last stage.
SIGNIFICANCE AND USE
4.1 This practice considers inspection procedures that may involve multiple-stage sampling, where at each stage one can decide to accept or to continue sampling, and the decision to reject is deferred until the last stage.  
4.1.1 At each stage there are one or more acceptance criteria on the test results; for example, limits on each individual test result, or limits on statistics based on the sample of test results, such as the average, standard deviation, or coefficient of variation (relative standard deviation).  
4.2 The methodology in this practice defines an acceptance region for a set of test results from the sampled population such that, at a prescribed confidence level, the probability that a sample from the population will pass the acceptance procedure is greater than or equal to a prespecified lower bound.  
4.2.1 Having test results fall in the acceptance region is not equivalent to passing the acceptance procedure, but provides assurance that a sample would pass the acceptance procedure with a specified probability.  
4.2.2 This information can be used for process demonstration, validation of test methods, and qualification of instruments, processes, and materials.  
4.2.3 This information can be used for lot release (acceptance), but the lower bound may be conservative in some cases.  
4.2.4 If the results are to be applied to future test results from the same process, then it is assumed that the process is stable and predictable. If this is not the case then there can be no guarantee that the probability estimates would be valid predictions of future process performance.  
4.3 This methodology was originally developed (1-4)3 for use in two specific quality characteristics of drug products in the pharmaceutical industry but will be applicable for acceptance procedures in all industries.  
4.4 Mathematical derivations would be required that are specific to the individual criteria of each test.
SCOPE
1.1 This practice provides a general methodology for evaluating single-stage or multiple-stage acceptance procedures which involve a quality characteristic measured on a numerical scale. This methodology computes, at a prescribed confidence level, a lower bound on the probability of passing an acceptance procedure, using estimates of the parameters of the distribution of test results from a sampled population.  
1.2 For a prescribed lower probability bound, the methodology can also generate an acceptance limit table, which defines a set of test method outcomes (for example, sample averages and standard deviations) that would pass the acceptance procedure at a prescribed confidence level.  
1.3 This approach may be used for demonstrating compliance with in-process, validation, or lot-release specifications.  
1.4 The system of units for this practice is not specified.  
1.5 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.6 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the De...

ABSTRACT
The purpose of this guide is to provide guidelines for identifying the elements that comprise the test result of a test method and to illustrate how these elements combine into the test result. It covers the types of measurement scales used for expressing observations and test results. This guide provides information on the construction of test results from more elemental measurements.
This guide covers a test method that may have three distinct stages: the direct measurement or observation of dimensions or properties, or the occurrence of an event; the arithmetical combination of observed values to obtain a single determination; and the arithmetical combination of a number of determinations to obtain the test result of a test method. This guide also explains that for a quantitative test method, a test determination may be described as the process of calculating from one or more observations a property of a single test specimen, or as the value obtained from the process.
SIGNIFICANCE AND USE
4.1 All test methods have an output in the form of a test result. This guide provides information on the construction of test results from more elemental measurements.  
4.2 A well defined test result is necessary before any precision statements can be made about the test method.  
4.2.1 Form and Style for ASTM Standards,2 Section A21, requires that every test method shall contain a statement regarding its precision, preferably as a result of an interlaboratory test program. Reporting of such studies is described in Practice E177, which illustrates the development of test results from observations and test determinations.  
4.2.2 Precision statements for ASTM test methods are applicable to test results. They are not applicable to test determinations or observations, unless specifically and clearly indicated otherwise.
SCOPE
1.1 The purpose of this guide is to provide guidelines for identifying the elements that comprise the test result of a test method and to illustrate how these elements combine into the test result.  
1.2 Types of measurement scales used for expressing observations and test results are discussed.  
1.3 No system of units is specified in this standard.  
1.4 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled. This practice restricts itself to designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results. Conducting a ruggedness test requires making systematic changes in the variables, called factors, and then observing the subsequent effect of those changes upon the test result of each run.
This practice recommends statistically designed experiments involving two levels of multiple factors. The steps to be conducted include identification of relevant factors; selection of appropriate levels (two for each factor) to be used in experiment runs; display of treatment combinations in cyclic shifted order, which assigns factors and levels to runs; execution of runs arranged in a random order; statistical analysis to determine the effect of factors on the test method results; and possible revision of the test method as needed.
SIGNIFICANCE AND USE
5.1 A ruggedness test is a special application of a statistically designed experiment. It is generally carried out when it is desirable to examine a large number of possible factors to determine which of these factors might have the greatest effect on the outcome of a test method. Statistical design enables more efficient and cost effective determination of the factor effects than would be achieved if separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed for evaluating quantitative test methods.  
5.2 In ruggedness testing, the two levels for each factor are chosen to use moderate separations between the high and low settings. In general, the size of effects, and the likelihood of interactions between the factors, will increase with increased separation between the high and low settings of the factors.  
5.3 Ruggedness testing is usually done within a single laboratory on uniform material, so the effects of changing only the factors are measured. The results may then be used to assist in determining the degree of control required of factors described in the test method.  
5.4 Ruggedness testing is part of the validation phase of developing a standard test method as described in Guide E1488. It is preferred that a ruggedness test precedes an interlaboratory (round robin) study.
SCOPE
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.  
1.2 This practice restricts itself to designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.  
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.5 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical ...

SIGNIFICANCE AND USE
4.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177. Knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values (such as specification limits) or between data sources (different laboratories, instruments, etc.).  
4.1.1 When a test method is applied to a large number of portions of a material that are as nearly alike as possible, the test results obtained will not all have the same value. A measure of the degree of agreement among these test results describes the precision of the test method for that material. Numerical measures of the variability between such test results provide inverse measures of the precision of the test method. Greater variability implies smaller (that is, poorer) precision and larger imprecision.  
4.1.2 Precision is reported as a standard deviation, coefficient of variation (relative standard deviation), variance, or a precision limit (a data range indicating no statistically significant difference between test results).  
4.1.3 This practice is designed only to estimate the precision of a test method. However, when accepted reference values are available for the property levels, the test result data obtained according to this practice may be used in estimating the bias of the test method. For a discussion of bias estimation and the relationships between precision, bias, and accuracy, see Practice E177.  
4.2 The procedures presented in this practice consist of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data.  
4.2.1 The planning phase includes forming the ILS task group, the study design, selection and number of participating laborator...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements.  
1.4.1 This practice does not cover methods in which the measurement is a categorization; however, for many practical purposes categorical outcomes can be scored, such as zero-one scoring for binary measurements or as integers, ranks for example, for well-ordered categories and then the test result can be defined as an average, or other summary statistic, of several individual scores.  
1.5 This standard may involve hazardous materials, operations, and equipment. This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.6 This international standard was developed...

ABSTRACT
This guide identifies statistical procedures for use in developing new test methods or revising or evaluating existing test methods, or both. It also cites statistical procedures especially useful in the application of test methods. This standard recommends what approaches may be taken and indicates which standards may be used to perform such assessments.
This standard introduces a series of phases which are recommended to be considered during the life cycle of a test method. These begin with a design phase where the standard is initially prepared. A development phase involves a variety of experiments that allow further refinement and understanding of how the test method performs within a laboratory. In an evaluation phase the test method is then examined by way of inter-laboratory studies resulting in precision and bias statistics which are published in the standard. Finally, the test method is subject to a monitoring phase.
SIGNIFICANCE AND USE
4.1 The creation of a standardized test method generally follows a series of steps from inception to approval and ongoing use. In all such stages there are questions of how well the test method performs.  
4.1.1 Assessments of a new or existing test method generally involve statistical planning and analysis. This standard recommends what approaches may be taken and indicates which standards may be used to perform such assessments.  
4.2 This standard introduces a series of phases which are recommended to be considered during the life cycle of a test method as depicted in Fig. 1. These begin with a design phase where the standard is initially prepared. A development phase involves a variety of experiments that allow further refinement and understanding of how the test method performs within a laboratory. In an evaluation phase the test method is then examined by way of interlaboratory studies resulting in precision and bias statistics which are published in the standard. Finally, the test method is subject to a monitoring phase.
FIG. 1 Sequence of Steps  
4.3 All ASTM test methods are required to include statements on precision and bias.3  
4.4 Since ASTM began to require all test methods to have precision and bias statements that are based on interlaboratory test methods, there has been increased concern regarding what statistical experiments and procedures to use during the development of the test methods. Although there exists a wide range of statistical procedures, there is a small group of generally accepted techniques that are beneficial to follow. This guide is designed to provide a brief overview of these procedures and to suggest an appropriate sequence of carrying out these procedures.  
4.5 Statistical procedures often result in interpretations that are not absolutes. Sometimes the information obtained may be inadequate or incomplete, which may lead to additional questions and the need for further experimentation. Information outside the data is al...
SCOPE
1.1 This guide identifies statistical procedures for use in developing new test methods or revising or evaluating existing test methods, or both.  
1.2 This guide also cites statistical procedures especially useful in the application of test methods.  
1.3 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

ABSTRACT
This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources to determine if their true means or variances differ by no more than predetermined limits. This standard provides guidance on experiments and statistical methods needed to demonstrate that the test results from a modified testing process are equivalent to those from the current testing process, where equivalence is defined as agreement within a prescribed limit, termed an equivalence limit.
SIGNIFICANCE AND USE
4.1 Laboratories conducting routine testing have a continuing need to make improvements in their testing processes. In these situations it must be demonstrated that any changes will neither cause an undesirable shift in the test results from the current testing process nor substantially affect a performance characteristic of the test method. This standard provides guidance on experiments and statistical methods needed to demonstrate that the test results from a modified testing process are equivalent to those from the current testing process, where equivalence is defined as agreement within a prescribed limit, termed an equivalence limit.  
4.1.1 The equivalence limit, which represents a worst-case difference or ratio, is determined prior to the equivalence test and its value is usually set by consensus among subject-matter experts.  
4.1.2 Examples of modifications to the testing process include, but are not limited, to the following:  
(1) Changes to operating levels in the steps of the test method procedure,
(2) Installation of new instruments, apparatus, or sources of reagents and test materials,
(3) Evaluation of new personnel performing the testing, and
(4) Transfer of testing to a new location.  
4.1.3 Examples of performance characteristics directly applicable to the test method include bias, precision, sensitivity, specificity, linearity, and range. Additional characteristics are test cost and elapsed time needed to conduct the test procedure.  
4.2 Equivalence testing is performed by a designed experiment that generates test results from the modified and current testing procedures on the same types of materials that are routinely tested. The design of the experiment depends on the type of equivalence needed as discussed below. Experiment design and execution for various objectives is discussed in Section 5.  
4.2.1 Means equivalence is concerned with a potential shift in the mean test result in either direction due to a modification in the tes...
SCOPE
1.1 This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources to determine if their true means or variances differ by no more than predetermined limits.  
1.2 Applications include (1) equivalence testing for bias against an accepted reference value, (2) determining means equivalence of two test methods, test apparatus, instruments, reagent sources, or operators within a laboratory or equivalence of two laboratories in a method transfer, and (3) determining non-inferiority of a modified test procedure versus a current test procedure with respect to a performance characteristic.  
1.3 The guidance in this standard applies to experiments conducted on a single material at a given level of the test result or on multiple materials covering a range of selected test results.  
1.4 Guidance is given for determining the amount of data required for an equivalence trial. The control of risks associated with the equivalence decision is discussed.  
1.5 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations...

SIGNIFICANCE AND USE
5.1 A ruggedness test is a special application of a statistically designed experiment. It is generally carried out when it is desirable to examine a large number of possible factors to determine which of these factors might have the greatest effect on the outcome of a test method. Statistical design enables more efficient and cost effective determination of the factor effects than would be achieved if separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed for evaluating quantitative test methods.  
5.2 In ruggedness testing, the two levels for each factor are chosen to use moderate separations between the high and low settings. In general, the size of effects, and the likelihood of interactions between the factors, will increase with increased separation between the high and low settings of the factors.  
5.3 Ruggedness testing is usually done within a single laboratory on uniform material, so the effects of changing only the factors are measured. The results may then be used to assist in determining the degree of control required of factors described in the test method.  
5.4 Ruggedness testing is part of the validation phase of developing a standard test method as described in Guide E1488. It is preferred that a ruggedness test precedes an interlaboratory (round robin) study.
SCOPE
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.  
1.2 This practice restricts itself to designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.  
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.5 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

SIGNIFICANCE AND USE
5.1 A ruggedness test is a special application of a statistically designed experiment. It is generally carried out when it is desirable to examine a large number of possible factors to determine which of these factors might have the greatest effect on the outcome of a test method. Statistical design enables more efficient and cost effective determination of the factor effects than would be achieved if separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed for evaluating quantitative test methods.  
5.2 In ruggedness testing, the two levels for each factor are chosen to use moderate separations between the high and low settings. In general, the size of effects, and the likelihood of interactions between the factors, will increase with increased separation between the high and low settings of the factors.  
5.3 Ruggedness testing is usually done within a single laboratory on uniform material, so the effects of changing only the factors are measured. The results may then be used to assist in determining the degree of control required of factors described in the test method.  
5.4 Ruggedness testing is part of the validation phase of developing a standard test method as described in Guide E1488. It is preferred that a ruggedness test precedes an interlaboratory (round robin) study.
SCOPE
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.  
1.2 This practice restricts itself to designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.  
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.5 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

SIGNIFICANCE AND USE
4.1 Laboratories conducting routine testing have a continuing need to make improvements in their testing processes. In these situations it must be demonstrated that any changes will not cause an undesirable shift in the test results from the current testing process nor substantially affect a performance characteristic of the test method. This standard provides guidance on experiments and statistical methods needed to demonstrate that the test results from a modified testing process are equivalent to those from the current testing process, where equivalence is defined as agreement within a prescribed limit, termed an equivalence limit.  
4.1.1 Examples of modifications to the testing process include, but are not limited, to the following:  
(1) Changes to operating levels in the steps of the test method procedure,
(2) Installation of new instruments, apparatus, or sources of reagents and test materials,  
(3) Evaluation of new personnel performing the testing, and
(4) Transfer of testing to a new location.  
4.1.2 The equivalence limit, which represents a worst-case difference, is determined prior to the equivalence test and its value is usually set by consensus among subject-matter experts.  
4.2 Two principal types of equivalence are covered in the practice, means equivalence and non-inferiority. Means equivalence implies that a sustained shift in test results between the modified and current testing processes refers to an absolute difference, meaning differences in either direction from zero. Non-inferiority is concerned with a difference only in the direction of an inferior outcome in a performance characteristic of the modified testing procedure versus the current testing procedure.  
4.2.1 Equivalence testing is performed by an experiment that generates test results from the modified and current testing procedures on the same materials that are routinely tested. An exception is bias equivalence where the experiment consists of conducting multiple te...
SCOPE
1.1 This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources to determine if their true means or variances differ by no more than predetermined limits.  
1.2 Applications include (1) equivalence testing for bias against an accepted reference value, (2) determining means equivalence of two test methods, test apparatus, instruments, reagent sources, or operators within a laboratory or equivalence of two laboratories in a method transfer, and (3) determining non-inferiority of a modified test procedure versus a current test procedure with respect to a performance characteristic.  
1.3 The guidance in this standard applies only to experiments conducted on a single material at a given level of the test result.  
1.4 Guidance is given for determining the amount of data required for an equivalence trial. The control of risks associated with the equivalence decision is discussed.  
1.5 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

ABSTRACT
This practice describes methods for the statistical analysis of laboratory results obtained from interlaboratory proficiency testing programs. As in accordance with Practice E1301, proficiency testing is the use of inter-laboratory comparisons for the determination of laboratory testing or measurement performance. The methods provide direction for assessing and categorizing the performance of individual laboratories based on the relative likelihood of occurrence of their test results, and for determining estimates of testing variation associated with repeatability and reproducibility. Assumptions are that a majority of the participating laboratories execute the test method properly and that samples are of sufficient homogeneity that the testing results represent results obtained from each laboratory testing essentially the same material. Each laboratory receives the same instructions or protocol.
This practice is specifically designed to describe simple robust statistical methods for use in proficiency testing programs.
SIGNIFICANCE AND USE
5.1 This practice is specifically designed to describe simple robust statistical methods for use in proficiency testing programs.  
5.2 Proficiency testing programs can use the methods in this practice for the purpose of comparing testing results obtained from a group of participating laboratories. The laboratory comparisons can then be used for evaluation of individual laboratory performance.  
5.3 In addition, the data obtained in proficiency testing programs may contain information regarding repeatability (within-lab) and reproducibility (between-lab) testing variation. Repeatability information is possible only if the program uses more than one sample. See Method B. Proficiency testing programs often have a greater number of participants than might be available for conducting an interlaboratory study to determine the precision of a test method (such as described in Practice E691). Precision estimates obtained for the larger number of participants in a proficiency testing program, along with the corresponding wider variation of test conditions, can provide useful information to standards developers regarding the precision of test results that can be expected for a test method when in actual use in the general testing community.  
5.4 To estimate the precision of a test method, the participants must use the same test method to obtain their test results, and testing must be performed under the conditions required for repeatability and reproducibility. The precision estimates are applicable to the property levels and material types included in the testing program. The precision of a test method may vary considerably for different material types and at different property levels.  
5.5 This practice may be useful to proficiency testing program administrators and provides examples of statistical methods along with explanations of some of the advantages of the suggested methods of analysis. The analyses resulting from the application of methods described in this pra...
SCOPE
1.1 This practice describes methods for the statistical analysis of laboratory results obtained from interlaboratory proficiency testing programs. As in accordance with Practice E1301, proficiency testing is the use of interlaboratory comparisons for the determination of laboratory testing or measurement performance. Conversely, collaborative study (or collaborative trial) is the use of interlaboratory comparisons for the determination of the precision of a test method, as covered by Practice E691.  
1.1.1 Method A covers testing programs using single test results obtained by testing a single sample (each laboratory submits a single test result).  
1.1.2 Method B covers testing programs using paired test results obtained by testing two samples (each laboratory submits one test result for each of the two samples). The two samples should be of the same material or two materials similar enoug...

SIGNIFICANCE AND USE
4.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177. Knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values (such as specification limits) or between data sources (different laboratories, instruments, etc.).  
4.1.1 When a test method is applied to a large number of portions of a material that are as nearly alike as possible, the test results obtained will not all have the same value. A measure of the degree of agreement among these test results describes the precision of the test method for that material. Numerical measures of the variability between such test results provide inverse measures of the precision of the test method. Greater variability implies smaller (that is, poorer) precision and larger imprecision.  
4.1.2 Precision is reported as a standard deviation, coefficient of variation (relative standard deviation), variance, or a precision limit (a data range indicating no statistically significant difference between test results).  
4.1.3 This practice is designed only to estimate the precision of a test method. However, when accepted reference values are available for the property levels, the test result data obtained according to this practice may be used in estimating the bias of the test method. For a discussion of bias estimation and the relationships between precision, bias, and accuracy, see Practice E177.  
4.2 The procedures presented in this practice consist of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data.  
4.2.1 The planning phase includes forming the ILS task group, the study design, selection and number of participating laborator...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements.  
1.4.1 This practice does not cover methods in which the measurement is a categorization; however, for many practical purposes categorical outcomes can be scored, such as zero-one scoring for binary measurements or as integers, ranks for example, for well-ordered categories and then the test result can be defined as an average, or other summary statistic, of several individual scores.  
1.5 This standard may involve hazardous materials, operations, and equipment. This standard does not purport to address all of the safety problems associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

SIGNIFICANCE AND USE
4.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177. Knowledge of the test method precision is useful in commerce and in technical work when comparing test results against standard values (such as specification limits) or between data sources (different laboratories, instruments, etc.).  
4.1.1 When a test method is applied to a large number of portions of a material that are as nearly alike as possible, the test results obtained will not all have the same value. A measure of the degree of agreement among these test results describes the precision of the test method for that material. Numerical measures of the variability between such test results provide inverse measures of the precision of the test method. Greater variability implies smaller (that is, poorer) precision and larger imprecision.  
4.1.2 Repeatability and Reproducibility—These two terms deal with the variability of test results obtained under specified laboratory conditions and represent the two extremes of test method precision. Repeatability concerns the variability between independent test results obtained within a single laboratory in the shortest practical period of time by a single operator with a specific set of test apparatus using test specimens (or test units) taken at random from a single quantity of homogeneous material obtained or prepared for the ILS. Reproducibility deals with the variability between single test results obtained in different laboratories, each of which has applied the test method to test specimens (or test units) taken at random from a single quantity of homogeneous material obtained or prepared for the ILS.  
4.1.2.1 Repeatability Conditions—The single-operator, single-set-of-apparatus requirement means that for a partic...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements.  
1.4.1 This practice does not cover methods in which the measurement is a categorization; however, for many practical purposes categorical outcomes can be scored, such as zero-one scoring for binary measurements or as integers, ranks for example, for well-ordered categories and then the test result can be defined as an average, or other summary statistic, of several individual scores.  
1.5 This standard may involve hazardous materials, operations, and equipment. This standard does not purport to address all of the safety problems associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

SIGNIFICANCE AND USE
4.1 Laboratories conducting routine testing have a continuing need to evaluate test result bias, to evaluate changes for improving the test process performance, or to validate the transfer of a test method to a new location or apparatus. In all situations it must be demonstrated that any bias or innovation will have negligible effect on test results for a characteristic of a material. This standard provides statistical methods to confirm that the mean test results from a testing process are equivalent to those from a reference standard or another testing process, where equivalence is defined as agreement within prescribed limits, termed equivalence limits.  
4.1.1 The intra-laboratory applications in this practice include, but are not limited to, the following:  
(1) Evaluating the bias of a test method with respect to a certified reference material,
(2) Evaluating bias due to a minor change in a test method procedure,  
(3) Qualifying new instruments, apparatus, or operators in a laboratory, and
(4) Qualifying new sources of reagents or other materials used in the test procedure.  
4.1.2 This practice also supports evaluating systematic differences in a method transfer from a developing laboratory to a receiving laboratory.  
4.2 This practice currently deals only with the equivalence of population means. In this standard, a population refers to a hypothetical set of test results arising from a stable testing process that measures a characteristic of a single material.
Note 1: The equivalence concept can also apply to population parameters other than means, such as precision, stated as variances, standard deviations, or relative standard deviations (coefficients of variation), linearity, sensitivity, specificity, etc.  
4.3 The data analysis for equivalence testing of population means in this practice uses a statistical methodology termed the two one-sided tests (TOST) procedure which shall be described in detail in this standard (see X1.1). The TOST pr...
SCOPE
1.1 This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources to determine if their true means differ by no more than predetermined limits.  
1.2 Applications include (1) equivalence testing for bias against an accepted reference value, (2) determining equivalence of two test methods, test apparatus, instruments, reagent sources, or operators within a laboratory, and (3) equivalence of two laboratories in a method transfer.  
1.3 The current guidance in this standard applies only to experiments conducted on a single material. Guidance is given for determining the amount of data required for an equivalence trial.  
1.4 The statistical methodology for determining equivalence used is the two one-sided tests (TOST) procedure. The control of risks associated with the equivalence decision is discussed.  
1.5 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

ABSTRACT
This guide covers key elements of an evaluation of a laboratory’s measurement practices and the statistical analysis of the resulting data, as well as addresses an evaluation that covers a broad range of in-house quality measurements, some of which may be directly related to accreditation requirements. It provides an overview of the documentation needed for verification and monitoring of the practices used in the laboratory for measurement. In addition, it guides the user in verifying that the extent of documentation and the quality of statistical evaluations performed on the data being generated is sufficient. This guide is also intended to provide guidance for laboratory quality managers, accrediting bodies and assessors in evaluating the measurement practices of a laboratory and statistically analyzing the resulting data from these practices.
This guide provides the purpose of evaluating measurement practices and the statistical analysis of the resulting data. Data generated from the measurement practices of a laboratory are evaluated to determine its bias and precision performance, and to determine if the laboratory correctly and efficiently analyzes and reacts to its own data.
SIGNIFICANCE AND USE
4.1 This guide is intended to provide guidance for laboratory quality managers, accrediting bodies and assessors in evaluating the measurement practices of a laboratory, the protocol for statistically analyzing the resulting data from these practices, and the statistical results from these practices.  
4.2 This guide is generic in the sense that it covers the entire range of in-house quality measurement practices found in a testing laboratory, and the results of the described evaluation may be used by accrediting agencies for assessment purposes to determine whether their requirements can be satisfied through the laboratory's existing quality data program.  
4.3 It is not the intent of this guide to serve as sole criterion for evaluating and accrediting laboratories. Evaluation of measurement practices is only one aspect in a comprehensive quality program.
SCOPE
1.1 This guide covers key elements of an evaluation of a laboratory's measurement practices and the statistical analysis of the resulting data. This guide addresses an evaluation that covers a broad range of in-house quality measurements, some of which may be directly related to accreditation requirements.  
1.2 This guide provides an overview of the documentation needed for verification and monitoring of the practices used in the laboratory for measurement. In addition, it guides the user in verifying that the extent of documentation and the quality of statistical evaluations performed on the data being generated is sufficient. The user is advised to fully document all work covered by the scope of this guide as a general principle of laboratory practice and for audit purposes, whether internal or external.  
1.3 This guide is not designed to be exhaustive for all aspects of work realized under its scope. The user is encouraged to thoroughly realize (achieve in practice) the principles set forth in this guide, consulting other relevant standards and industry documents when appropriate.

SIGNIFICANCE AND USE
4.1 Laboratories conducting routine testing have a continuing need to evaluate test result bias, to evaluate changes for improving the test process performance, or to validate the transfer of a test method to a new location or apparatus. In all situations it must be demonstrated that any bias or innovation will have negligible effect on test results for a characteristic of a material. This standard provides statistical methods to confirm that the mean test results from a testing process are equivalent to those from a reference standard or another testing process, where equivalence is defined as agreement within prescribed limits, termed equivalence limits.  
4.1.1 The intra-laboratory applications in this practice include, but are not limited to, the following:  
(1) Evaluating the bias of a test method with respect to a certified reference material,
(2) Evaluating bias due to a minor change in a test method procedure,  
(3) Qualifying new instruments, apparatus, or operators in a laboratory, and
(4) Qualifying new sources of reagents or other materials used in the test procedure.  
4.1.2 This practice also supports evaluating systematic differences in a method transfer from a developing laboratory to a receiving laboratory.  
4.2 This practice currently deals only with the equivalence of population means. In this standard, a population refers to a hypothetical set of test results arising from a stable testing process that measures a characteristic of a single material.
Note 1: The equivalence concept can also apply to population parameters other than means, such as precision, stated as variances, standard deviations, or relative standard deviations (coefficients of variation), linearity, sensitivity, specificity, etc.  
4.3 The data analysis for equivalence testing of population means in this practice uses a statistical methodology termed the “Two one-sided t-test” (TOST) procedure which shall be described in detail in this standard (see X1.1). The TOST...
SCOPE
1.1 This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources to determine if their true means are similar within predetermined limits.  
1.2 Applications include (1) equivalence testing for bias against an accepted reference value, (2) determining equivalence of two test methods, test apparatus, instruments, reagent sources, or operators within a laboratory, and (3) equivalence of two laboratories in a method transfer.  
1.3 The current guidance in this standard applies only to experiments conducted on a single material. Guidance is given for determining the amount of data required for an equivalence trial.  
1.4 The statistical methodology for determining equivalence used is the “Two one-sided t-test” (TOST). The control of risks associated with the equivalence decision is discussed.  
1.5 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

SIGNIFICANCE AND USE
4.1 This practice considers inspection procedures that may involve multiple-stage sampling, where at each stage one can decide to accept or to continue sampling, and the decision to reject is deferred until the last stage.  
4.1.1 At each stage there are one or more acceptance criteria on the test results; for example, limits on each individual test result, or limits on statistics based on the sample of test results, such as the average, standard deviation, or coefficient of variation (relative standard deviation).  
4.2 The methodology in this practice defines an acceptance region for a set of test results from the sampled population such that, at a prescribed confidence level, the probability that a sample from the population will pass the acceptance procedure is greater than or equal to a prespecified lower bound.  
4.2.1 Having test results fall in the acceptance region is not equivalent to passing the acceptance procedure, but provides assurance that a sample would pass the acceptance procedure with a specified probability.  
4.2.2 This information can be used for process demonstration, validation of test methods, and qualification of instruments, processes, and materials.  
4.2.3 This information can be used for lot release (acceptance), but the lower bound may be conservative in some cases.  
4.2.4 If the results are to be applied to future test results from the same process, then it is assumed that the process is stable and predictable. If this is not the case then there can be no guarantee that the probability estimates would be valid predictions of future process performance.  
4.3 This methodology was originally developed (1-4)3 for use in two specific quality characteristics of drug products in the pharmaceutical industry but will be applicable for acceptance procedures in all industries.  
4.4 Mathematical derivations would be required that are specific to the individual criteria of each test.
SCOPE
1.1 This practice provides a general methodology for evaluating single-stage or multiple-stage acceptance procedures which involve a quality characteristic measured on a numerical scale. This methodology computes, at a prescribed confidence level, a lower bound on the probability of passing an acceptance procedure, using estimates of the parameters of the distribution of test results from a sampled population.  
1.2 For a prescribed lower probability bound, the methodology can also generate an acceptance limit table, which defines a set of test method outcomes (for example, sample averages and standard deviations) that would pass the acceptance procedure at a prescribed confidence level.  
1.3 This approach may be used for demonstrating compliance with in-process, validation, or lot-release specifications.  
1.4 The system of units for this practice is not specified.  
1.5 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

SIGNIFICANCE AND USE
4.1 This practice considers inspection procedures that may involve multiple-stage sampling, where at each stage one can decide to accept or to continue sampling, and the decision to reject is deferred until the last stage.  
4.1.1 At each stage there are one or more acceptance criteria on the test results; for example, limits on each individual test result, or limits on statistics based on the sample of test results, such as the average, standard deviation, or coefficient of variation (relative standard deviation).  
4.2 The methodology in this practice defines an acceptance region for a set of test results from the sampled population such that, at a prescribed confidence level, the probability that a sample from the population will pass the acceptance procedure is greater than or equal to a prespecified lower bound.  
4.2.1 Having test results fall in the acceptance region is not equivalent to passing the acceptance procedure, but provides assurance that a sample would pass the acceptance procedure with a specified probability.  
4.2.2 This information can be used for process demonstration, validation of test methods, and qualification of instruments, processes, and materials.  
4.2.3 This information can be used for lot release (acceptance), but the lower bound may be conservative in some cases.  
4.2.4 If the results are to be applied to future test results from the same process, then it is assumed that the process is stable and predictable. If this is not the case then there can be no guarantee that the probability estimates would be valid predictions of future process performance.  
4.3 This methodology was originally developed (1-4)3 for use in two specific quality characteristics of drug products in the pharmaceutical industry but will be applicable for acceptance procedures in all industries.  
4.4 Mathematical derivations would be required that are specific to the individual criteria of each test.
SCOPE
1.1 This practice provides a general methodology for evaluating single-stage or multiple-stage acceptance procedures which involve a quality characteristic measured on a numerical scale. This methodology computes, at a prescribed confidence level, a lower bound on the probability of passing an acceptance procedure, using estimates of the parameters of the distribution of test results from a sampled population.  
1.2 For a prescribed lower probability bound, the methodology can also generate an acceptance limit table, which defines a set of test method outcomes (for example, sample averages and standard deviations) that would pass the acceptance procedure at a prescribed confidence level.  
1.3 This approach may be used for demonstrating compliance with in-process, validation, or lot-release specifications.  
1.4 The system of units for this practice is not specified.  
1.5 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

SIGNIFICANCE AND USE
4.1 All test methods have an output in the form of a test result. This guide provides information on the construction of test results from more elemental measurements.  
4.2 A well defined test result is necessary before any precision statements can be made about the test method.  
4.2.1 Form and Style for ASTM Standards, Section A21, requires that every test method shall contain a statement regarding its precision, preferably as a result of an interlaboratory test program. Reporting of such studies is described in Practice E177, which illustrates the development of test results from observations and test determinations.  
4.2.2 Precision statements for ASTM test methods are applicable to test results. They are not applicable to test determinations or observations, unless specifically and clearly indicated otherwise.
SCOPE
1.1 The purpose of this guide is to provide guidelines for identifying the elements that comprise the test result of a test method and to illustrate how these elements combine into the test result.  
1.2 Types of measurement scales used for expressing observations and test results are discussed.  
1.3 No system of units is specified in this standard.

SIGNIFICANCE AND USE
5.1 A ruggedness test is a special application of a statistically designed experiment. It is generally carried out when it is desirable to examine a large number of possible factors to determine which of these factors might have the greatest effect on the outcome of a test method. Statistical design enables more efficient and cost effective determination of the factor effects than would be achieved if separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed for evaluating quantitative test methods.  
5.2 In ruggedness testing, the two levels for each factor are chosen to use moderate separations between the high and low settings. In general, the size of effects, and the likelihood of interactions between the factors, will increase with increased separation between the high and low settings of the factors.  
5.3 Ruggedness testing is usually done within a single laboratory on uniform material, so the effects of changing only the factors are measured. The results may then be used to assist in determining the degree of control required of factors described in the test method.  
5.4 Ruggedness testing is part of the validation phase of developing a standard test method as described in Guide E1488. It is preferred that a ruggedness test precedes an interlaboratory (round robin) study.
SCOPE
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.  
1.2 This practice restricts itself to designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.  
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

SIGNIFICANCE AND USE
4.1 Part A of the “Blue Book,” Form and Style for ASTM Standards, requires that all test methods include statements of precision and bias. This practice discusses these two concepts and provides guidance for their use in statements about test methods.  
4.2 Precision—A statement of precision allows potential users of a test method to assess in general terms the test method's usefulness with respect to variability in proposed applications. A statement of precision is not intended to exhibit values that can be exactly duplicated in every user's laboratory. Instead, the statement provides guidelines as to the magnitude of variability that can be expected between test results when the method is used in one, or in two or more, reasonably competent laboratories. For a discussion of precision, see 8.1.  
4.3 Bias—A statement of bias furnishes guidelines on the relationship between a set of typical test results produced by the test method under specific test conditions and a related set of accepted reference values (see 9.1).  
4.3.1 An alternative term for bias is trueness, which has a positive connotation, in that greater bias is associated with less favorable trueness. Trueness is the systematic component of accuracy.  
4.4 Accuracy—The term “accuracy,” used in earlier editions of Practice E177, embraces both precision and bias (see 9.3).
SCOPE
1.1 The purpose of this practice is to present concepts necessary to the understanding of the terms “precision” and “bias” as used in quantitative test methods. This practice also describes methods of expressing precision and bias and, in a final section, gives examples of how statements on precision and bias may be written for ASTM test methods.  
1.2 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory requirements prior to use.

ABSTRACT
The procedure presented in this practice consists of three basic steps: planning the interlaboratory study, guiding the testing phase of the study, and analyzing the test result data. The analysis utilizes tabular, graphical, and statistical diagnostic tools for evaluating the consistency of the data so that unusual values may be detected and investigated, and also includes the calculation of the numerical measures of precision of the test method pertaining to both within-laboratory repeatability and between-laboratory reproducibility.
Tests performed on presumably identical materials in presumably identical circumstances do not, in general, yield identical results. This is attributed to unavoidable random errors inherent in every test procedure; the factors that may influence the outcome of a test cannot all be completely controlled. In the practical interpretation of test data, this inherent variability has to be taken into account. For instance, the difference between a test result and some specified value may be within that which can be expected due to unavoidable random errors, in which case a real deviation from the specified value has not been demonstrated. Similarly, the difference between test results from two batches of material will not indicate a fundamental quality difference if the difference is no more than can be attributed to inherent variability in the test procedure. Many different factors (apart from random variations between supposedly identical specimens) may contribute to the variability in application of a test method, including: a the operator, b equipment used, c calibration of the equipment, and d environment (temperature, humidity, air pollution, etc.). It is considered that changing laboratories changes each of the above factors. The variability between test results obtained by different operators or with different equipment will usually be greater than between test results obtained by a single operator using the same equipment. The variability between test results taken over a long period of time even by the same operator will usually be greater than that obtained over a short period of time because of the greater possibility of changes in each of the above factors, especially the environment.
The general term for expressing the closeness of test results to the “true” value or the accepted reference value is accuracy. To be of practical value, standard procedures are required for determining the accuracy of a test method, both in terms of its bias and in terms of its precision. This practice provides a standard procedure for determining the precision of a test method. Precision, when evaluating test methods, is expressed in terms of two measurement concepts, repeatability and reproducibility. Under repeatability conditions the factors listed above are kept or remain reasonably constant and usually contribute only minimally to the variability. Under reproducibility conditions the factors are generally different (that is, they change from laboratory to laboratory) and usually contribute appreciably to the variability of test results. Thus, repeatability and reproducibility are two practical extremes of precision.
The repeatability measure, by excluding the factors a through d as contributing variables, is not intended as a mechanism for verifying the ability of a laboratory to maintain“ in-control” conditions for routine operational factors such as operator-to-operator and equipment differences or any effects of longer time intervals between test results. Such a control study is a separate issue for each laboratory to consider for itself, and is not a recommended part of an interlaboratory study.
The reproducibility measure (including the factors a through d as sources of variability) reflects what precision might be expected when random portions of a homogeneous sample are sent to random “in-control” laboratories.
To obtain reasonable estimates of repeatability and reprod...

SIGNIFICANCE AND USE
4.1 All test methods have an output in the form of a test result. This guide provides information on the construction of test results from more elemental measurements.  
4.2 A well defined test result is necessary before any precision statements can be made about the test method.  
4.2.1 Form and Style for ASTM Standards, Section A21, requires that every test method shall contain a statement regarding its precision, preferably as a result of an interlaboratory test program. Reporting of such studies is described in Practice E177, which illustrates the development of test results from observations and test determinations.  
4.2.2 Precision statements for ASTM test methods are applicable to test results. They are not applicable to test determinations or observations, unless specifically and clearly indicated otherwise.
SCOPE
1.1 The purpose of this guide is to provide guidelines for identifying the elements that comprise the test result of a test method and to illustrate how these elements combine into the test result.

SIGNIFICANCE AND USE
4.1 Laboratories conducting routine testing have a continuing need to evaluate test result bias, to evaluate changes for improving the test process performance, or to validate the transfer of a test method to a new location or apparatus. In all situations it must be demonstrated that any bias or innovation will have negligible effect on test results for a characteristic of a material. This standard provides statistical methods to confirm that the mean test results from a testing process are equivalent to those from a reference standard or another testing process, where equivalence is defined as agreement within prescribed limits, termed equivalence limits.  
4.1.1 The intra-laboratory applications in this practice include, but are not limited to, the following: (1) Evaluating the bias of a test method with respect to a certified reference material,(2) Evaluating bias due to a minor change in a test method procedure, (3) Qualifying new instruments, apparatus, or operators in a laboratory, and(4) Qualifying new sources of reagents or other materials used in the test procedure.  
4.1.2 This practice also supports evaluating bias in a method transfer from a developing laboratory to a receiving laboratory.  
4.2 This practice currently deals only with the equivalence of population means. In this standard, a population refers to a hypothetical set of test results arising from a stable testing process that measures a characteristic of a single material.Note 1—The equivalence concept can also apply to population parameters other than means, such as precision, stated as variances, standard deviations, or relative standard deviations (coefficients of variation), linearity, sensitivity, specificity, etc.  
4.3 The data analysis for equivalence testing of population means in this practice uses a statistical methodology termed the “Two one-sided t-test” (TOST) procedure which shall be described in detail in this standard (see X1.1). The TOST procedure will be adapted to the...
SCOPE
1.1 This practice provides statistical methodology for conducting equivalence testing on numerical data from two sources to determine if their true means are similar within predetermined limits.  
1.2 Applications include (1) equivalence testing for bias against an accepted reference value, (2) determining equivalence of two test methods, test apparatus, instruments, reagent sources, or operators within a laboratory, and (3) equivalence of two laboratories in a method transfer.  
1.3 The current guidance in this standard applies only to experiments conducted on a single material. Guidance is given for determining the amount of data required for an equivalence trial.  
1.4 The statistical methodology for determining equivalence used is the “Two one-sided t-test” (TOST). The control of risks associated with the equivalence decision is discussed.  
1.5 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this standard.  
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

SIGNIFICANCE AND USE
5.1 A ruggedness test is a special application of a statistically designed experiment. It is generally carried out when it is desirable to examine a large number of possible factors to determine which of these factors might have the greatest effect on the outcome of a test method. Statistical design enables more efficient and cost effective determination of the factor effects than would be achieved if separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed for evaluating quantitative test methods.  
5.2 In ruggedness testing, the two levels for each factor are chosen to use moderate separations between the high and low settings. In general, the size of effects, and the likelihood of interactions between the factors, will increase with increased separation between the high and low settings of the factors.  
5.3 Ruggedness testing is usually done within a single laboratory on uniform material, so the effects of changing only the factors are measured. The results may then be used to assist in determining the degree of control required of factors described in the test method.  
5.4 Ruggedness testing is part of the validation phase of developing a standard test method as described in Guide E1488. It is preferred that a ruggedness test precedes an interlaboratory (round robin) study.
SCOPE
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.  
1.2 This practice restricts itself to designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.  
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

SIGNIFICANCE AND USE
5.1 ASTM regulations require precision statements in all test methods in terms of repeatability and reproducibility. This practice may be used in obtaining the needed information as simply as possible. This information may then be used to prepare a precision statement in accordance with Practice E177.  
5.2 Test Method and Protocol—In this practice, the term “test method” is used both for the actual measurement process and for the written description of the process, while the term “protocol” is used for the directions given to the laboratories for conducting the ILS.  
5.3 Observations, Test Determinations and Test Results:  
5.3.1 A test method often has three distinct stages, the direct observation of dimensions or properties, the arithmetic combination of the observed values to obtain a test determination, and the arithmetic combination of a number of test determinations to obtain the test result of the test method. In the simplest of test methods a single direct observation is both the test determination and the test result. For example, the test method may require the measurement of the mass of a test specimen prepared in a prescribed way. Another test method may require the measurement of the area of the test specimen as well as the mass, and then direct that the mass be divided by the area to obtain the mass per unit area of the specimen. The whole process of measuring the mass and the area and calculating the mass per unit area is a test determination. If the test method specifies that only one test determination is to be made, then the test determination value is the test result of the test method. Some test methods require that several determinations be made and the values obtained be averaged or otherwise combined to obtain the test result of the test method. Averaging of several determinations is often used to reduce the effect of local variations of the property within the material.  
5.3.2 In this practice, the term “test determination” is used b...
SCOPE
1.1 This practice describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.  
1.2 This practice does not concern itself with the development of test methods but rather with gathering the information needed for a test method precision statement after the development stage has been successfully completed. The data obtained in the interlaboratory study may indicate, however, that further effort is needed to improve the test method.  
1.3 Since the primary purpose of this practice is the development of the information needed for a precision statement, the experimental design in this practice may not be optimum for evaluating materials, apparatus, or individual laboratories.  
1.4 Field of Application—This practice is concerned exclusively with test methods which yield a single numerical figure as the test result, although the single figure may be the outcome of a calculation from a set of measurements.  
1.4.1 This practice does not cover methods in which the measurement is a categorization, such as a go-no-go allocation (two categories) or a sorting scheme into two or more categories. For practical purposes, the discontinuous nature of measurements of these types may be ignored when a test result is defined as an average of several individual measurements. Then, this practice may be applicable, but caution is required and a statistician should be consulted.  
1.5 The information in this practice is arranged as follows:    
Section  
Scope  
1  
Referenced Documents  
2  
Terminology  
3  
Summary of Practice  
4  
Significance and Use  
5  
Planning the Interlaboratory Study (ILS)  
Section  
ILS Membership  
6  
Basic Design  
7  
Test Method  
8  
...

SIGNIFICANCE AND USE
4.1 Part A of the “Blue Book,” Form and Style for ASTM Standards, requires that all test methods include statements of precision and bias. This practice discusses these two concepts and provides guidance for their use in statements about test methods.  
4.2 Precision—A statement of precision allows potential users of a test method to assess in general terms the test method's usefulness with respect to variability in proposed applications. A statement of precision is not intended to exhibit values that can be exactly duplicated in every user's laboratory. Instead, the statement provides guidelines as to the magnitude of variability that can be expected between test results when the method is used in one, or in two or more, reasonably competent laboratories. For a discussion of precision, see 8.1.  
4.3 Bias—A statement of bias furnishes guidelines on the relationship between a set of typical test results produced by the test method under specific test conditions and a related set of accepted reference values (see 9.1).  
4.3.1 An alternative term for bias is trueness, which has a positive connotation, in that greater bias is associated with less favorable trueness. Trueness is the systematic component of accuracy.  
4.4 Accuracy—The term “accuracy,” used in earlier editions of Practice E177, embraces both precision and bias (see 9.3).
SCOPE
1.1 The purpose of this practice is to present concepts necessary to the understanding of the terms “precision” and “bias” as used in quantitative test methods. This practice also describes methods of expressing precision and bias and, in a final section, gives examples of how statements on precision and bias may be written for ASTM test methods.  
1.2 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory requirements prior to use.

SIGNIFICANCE AND USE
5.1 A ruggedness test is a special application of a statistically designed experiment. It is generally carried out when it is desirable to examine a large number of possible factors to determine which of these factors might have the greatest effect on the outcome of a test method. Statistical design enables more efficient and cost effective determination of the factor effects than would be achieved if separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed for evaluating quantitative test methods.  
5.2 In ruggedness testing, the two levels for each factor are chosen to use moderate separations between the high and low settings. In general, the size of effects, and the likelihood of interactions between the factors, will increase with increased separation between the high and low settings of the factors.  
5.3 Ruggedness testing is usually done within a single laboratory on uniform material, so the effects of changing only the factors are measured. The results may then be used to assist in determining the degree of control required of factors described in the test method.  
5.4 Ruggedness testing is part of the validation phase of developing a standard test method as described in Guide E1488. It is preferred that a ruggedness test precedes an interlaboratory (round robin) study.
SCOPE
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.  
1.2 This practice restricts itself to designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.  
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.

SIGNIFICANCE AND USE
4.1 This practice provides one way for a laboratory to develop data-based Type A estimates of uncertainty as referred to in Section A22 in Form and Style of ASTM Standards.  
4.2 Laboratories accredited under ISO/IEC 17025 are required to present uncertainty estimates for their test results. This practice provides procedures that use test results to develop uncertainty estimates for an individual laboratory.  
4.3 Generally, these test results will be from a single sample of stable and homogeneous material known as a control or check sample.  
4.4 The true value of the characteristic(s) of the control sample being measured will ordinarily be unknown. However, this methodology may also be used if the control sample is a reference material, in which case the test method bias may also be estimated and incorporated into the uncertainty estimate. Many test methods do not have true reference materials available to provide traceable chains of uncertainty estimation.  
4.5 This practice also allows for ongoing monitoring of the laboratory uncertainty. As estimates of the level of uncertainty change, possibly as contributions to uncertainty are identified and minimized, revision to the laboratory uncertainty will be possible.
SCOPE
1.1 This practice describes techniques for a laboratory to estimate the uncertainty of a test result using data from test results on a control sample. This standard provides one method for a laboratory to estimate Measurement Uncertainty in accordance with Section A22.3 in Form and Style of ASTM Standards.  
1.2 Uncertainty as defined by this practice applies to the capabilities of a single laboratory. Any estimate of uncertainty determined through the use of this practice applies only to the individual laboratory for which the data are presented.  
1.3 The laboratory uses a well defined and established test method in determining a series of test results. The uncertainty estimated using this practice only applies when the same test method is followed. The uncertainty only applies for the material types represented by the control samples, and multiple control samples may be needed, especially if the method has different precision for different sample types or response levels.  
1.4 The uncertainty estimate determined by this practice represents the intermediate precision of test results. This estimate seeks to quantify the total variation expected within a single laboratory using a single established test method while incorporating as many known sources of variation as possible.  
1.5 This practice does not establish error estimates (error budget) attributed to individual factors that could influence uncertainty.  
1.6 This practice describes the use of control charts to evaluate the data obtained and presents a special type of control chart to monitor the estimate of uncertainty.  
1.7 The system of units for this standard is not specified. Dimensional quantities in the standard are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.8 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.