ASTM E1169-89(1996)
(Guide)Standard Guide for Conducting Ruggedness Tests
Standard Guide for Conducting Ruggedness Tests
SCOPE
1.1 In studying a test method, it is necessary to consider the effect of environmental factors on the results obtained using the test method. If this effect is not considered, the results from the original developmental work on the test method may not be as accurate as expected.The purpose of a ruggedness test is to find the variables (experimental factors) that strongly influence the measurements provided by the test method, and to determine how closely these variables need to be controlled. Ruggedness tests do not determine the optimum conditions for the test method.
1.2 The experimental designs most often used in ruggedness testing are the so called "Plackett-Burman" designs (1). Other experimental designs also can be used. This guide, however, will restrict itself to Plackett-Burman designs with two levels per variable because these designs are particularly easy to use and are efficient in developing the information needed for improving test methods. The designs require the simultaneous change of the levels of all of the variables, and allow the determination of the separated effects of each of the variables on the measured results. In ruggedness tests the two levels for each variable are set so as not to be greatly different. For such situations, the calculated effect for any given variable is generally not greatly affected by changes in the level of any of the other variables. A detailed example involving glass electrode measurements of the pH of dilute acid solutions is used to illustrate ruggedness test procedures. A method is presented for evaluating the experimental uncertainties.
1.3 The information in this guide is arranged as follows: Section Scope 1 Summary of Guide 2 Significance and Use 3 Plackett-Burman Designs Applied to Ruggedness Tests 4 Plackett-Burman Design Calculations 5 Plackett-Burman Design Considerations 6 Interpretation of Results 7 Example 8 Testing Effects from Repeated (pH) Experiments 9 Controllable versus Uncontrollable Factors 10 Additional Information 11 Tables Figures Appendixes Additional Plackett-Burman Designs X1. Short-Cut Calculations X2. References
1.4 This standard may involve hazardous materials, operations, and equipment. This standard does not purport to address all of the safety problems associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.
General Information
Relations
Standards Content (Sample)
NOTICE: This standard has either been superseded and replaced by a new version or discontinued.
Contact ASTM International (www.astm.org) for the latest information.
Designation: E 1169 – 89 (Reapproved 1996) An American National Standard
AMERICAN SOCIETY FOR TESTING AND MATERIALS
100 Barr Harbor Dr., West Conshohocken, PA 19428
Reprinted from the Annual Book of ASTM Standards. Copyright ASTM
Standard Guide for
Conducting Ruggedness Tests
This standard is issued under the fixed designation E 1169; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (e) indicates an editorial change since the last revision or reapproval.
1. Scope
Testing Effects from Repeated (pH) Experiments 9
Controllable versus Uncontrollable Factors 10
1.1 In studying a test method, it is necessary to consider the
Additional Information 11
effect of environmental factors on the results obtained using the
Tables
Figures
test method. If this effect is not considered, the results from the
Appendixes
original developmental work on the test method may not be as
Additional Plackett-Burman Designs Appendix X1.
accurate as expected.The purpose of a ruggedness test is to find Short-Cut Calculations X1.3.
References
the variables (experimental factors) that strongly influence the
measurements provided by the test method, and to determine
1.4 This standard does not purport to address all of the
how closely these variables need to be controlled. Ruggedness
safety concerns, if any, associated with its use. It is the
tests do not determine the optimum conditions for the test
responsibility of the user of this standard to establish appro-
method.
priate safety and health practices and determine the applica-
1.2 The experimental designs most often used in ruggedness
bility of regulatory limitations prior to use.
testing are the so called “Plackett-Burman” designs (1). Other
2. Summary of Guide
experimental designs also can be used. This guide, however,
will restrict itself to Plackett-Burman designs with two levels
2.1 A ruggedness test is conducted by making systematic
per variable because these designs are particularly easy to use
changes in the variables associated with the test method and
and are efficient in developing the information needed for
observing the size of the associated changes in the test method
improving test methods. The designs require the simultaneous
results. Generally, the designs (systematic plans of experimen-
change of the levels of all of the variables, and allow the
tation) associated with ruggedness tests are taken from the field
determination of the separated effects of each of the variables
of statistics.
on the measured results. In ruggedness tests the two levels for
3. Significance and Use
each variable are set so as not to be greatly different. For such
situations, the calculated effect for any given variable is 3.1 The ruggedness test of a test method should precede an
generally not greatly affected by changes in the level of any of
interlaboratory study. The interlaboratory (round robin) study
the other variables. A detailed example involving glass elec- should be the final proof test for determining the precision of
trode measurements of the pH of dilute acid solutions is used
the test method. If a ruggedness test has not been run to
to illustrate ruggedness test procedures. A method is presented determine, and subsequently to restrict the allowable ranges of
for evaluating the experimental uncertainties.
the critical variables in the test method, then the precision from
1.3 The information in this guide is arranged as follows:
the round robin may be poor. It may not be known what went
Section wrong, or how to fix the test method. The ruggedness test, by
Scope 1
studying the influence of the test method variables and by
Summary of Guide 2
indicating the need for selective tightening of test method
Significance and Use 3
Plackett-Burman Designs Applied to Ruggedness Tests 4 specifications, helps avoid such situations. The use of rugged-
Plackett-Burman Design Calculations 5
ness tests encourages the orderly development of a test method.
Plackett-Burman Design Considerations 6
3.2 Ruggedness testing should be done within a single
Interpretation of Results 7
Example 8
laboratory so the effects of the variables are easier to see. Only
the effects of changes in the test method variables from high
levels to low levels need to be determined. Numerous variables
such as temperature, pressure, relative humidity, etc., may need
This guide is under the jurisdiction of ASTM Committee E-11 on Statistical
to be studied. The influences of these changes are best studied
Methods and is the direct responsibility of Subcommittee E11.20 on Test Method
Evaluation and Quality Control.
under the short-term, high-precision conditions found within a
Current edition approved Nov. 20, 1989. Published January 1990. Originally
single laboratory.
published as E 1168 – 87. Last previous edition E 1168 – 87.
The boldface numbers in parentheses refer to the list of references at the end of
this guide.
† Editorially corrected.
NOTICE: This standard has either been superseded and replaced by a new version or discontinued.
Contact ASTM International (www.astm.org) for the latest information.
E 1169
4. Plackett-Burman Designs Applied to Ruggedness Tests 4.4 A P-B design is constructed such that the four A( + ) and
the four A(−) terms will each be associated with an equal
4.1 A series of Plackett-Burman (P-B) designs are available
number of B( + ) and B(−) terms. The A effect is orthogonal to
for use with ruggedness tests for determining the effects of the
the B effect, that is, it is not affected by the B effect. In the P-B
test method variables (see 4.3 and Appendix X1). The effect for
design, all main effects (columns) are orthogonal to all other
each variable is calculated on the basis that a given change in
main effects (columns). This orthogonality of the main effects,
a variable from a high to a low level results in a fixed change
and the acceptance of possible contamination of estimates for
in the test result. It is common in ruggedness testing to assume
the main effects (by the interactions) are the major character-
that the observed effect of the simultaneous change of a
istics of most ruggedness tests. For many practical problems
number of variables can be described as the simple addition of
these characteristics are acceptable.
the fixed effects for each variable. It is also assumed that the
effect for each variable is independent of the effects of other
5. P-B Design Calculations
variables, that is, there are no coupled influences. The effects
5.1 The effect of any factor, such as A, is calculated as the
that are calculated on the basis of this assumption are called
average of the measurements made at the high level minus the
“main effects.” If a considerable lack of independence among
average of the measurements made at the low level, for
the effects of the variables is observed, the observer is than
example:
forced to recognize additional factors, which are called “inter-
actions.” The ruggedness test procedures for dealing with
(A~ 1 ! (A~2!
Effect A 5 2 5 ~2/N!@(A~1! 2 (A~2!# (1)
interactions are more complex, and are given in Refs (2) and N/2 N/2
(3). These more involved procedures, however, require addi-
Effect A 5 ~2/8! @~1.1 1 0.8 1 0.9 1 1.1!
tional measurements to develop information about the interac-
2 6.3 1 1.2 1 6.0 1 1.4!
~ #
tions. This guide is written only for evaluating main effects. 522.75.
4.2 P-B designs require that N must be an integer multiple
5.2 For the P-B design, the standard deviation for an effect,
of four, for example, 4, 8, 12, 16, etc. P-B designs for N
such as A, is easily derived by using Eq 1 along with the
measurements per replicate can be used to estimate up to N-1
standard deviation of a single measurement, s.
main effects. The calculated main effects, however, will be
s A 5 ~2/N! variance @(A~ 1 ! 2 (A~2!#
confounded (contaminated) with the interactions. If the inter- =
effect
actions are relatively small, then the user may be satisfied in
2 2
making only N ruggedness test measurements and obtaining 5 2/N Ns (2)
=~ !
slightly contaminated estimates for the N-1 main effects.
s A 5 2s/ N
=
effect
4.3 A P-B design for seven factors (A through G) and eight
The same equations for the P-B design apply when the
measurements is given in Fig. 1. This design is suitable for use
standard deviation s is replaced by its sample estimate, s,as
whenever an independent estimate of measurement variability
follows:
is available. Note that each column of the design contains an
equal number of plus ( + ) and minus (−) factor settings. A ( + )
s 5 2s/=N (3)
effect A
for a given factor indicates that the measurement is made with
Sections 7 and 9 present two methods for determining a
that factor set at the high level, and a (−) indicates the factor is
sample estimate of the standard deviation of a single measure-
to be at the low level. All seven factors are set for each
ment, s.
measurement (test result). The eight measurements should be
made in a random order. Typical test results are shown at the
6. P-B Design Considerations
far right of the design in Fig. 1. If slightly less than seven
6.1 Eq 3 shows that the standard deviation of an effect is
factors are being investigated, simply drop the “excess” col-
inversely proportional to N , the number of measurements
=
umns from the design. For such situations, the experimenter
made. The user may therefore be tempted to use large P-B
should consult a statistician to evaluate the measurement
designs. Practical experience, however, favors moderate size
variability. In this regard, Ref (1) (pp. 310 to 320) may be of
designs. Overly large designs require the correct setting of too
interest. The experimenter can, however, still use the tech-
many factors, and this increases the chance for blunders. In
niques described in Sections 6, 7, and 8 of this guide.
addition, large designs require more time to complete and other
factors not being considered in the design can change and
Factor
distort the results. The effects of incorrect factor settings and of
Test
shifting experimental conditions are propagated into all of the
AB C D E F G
Result
calculated results (see Eq 1). The (N 5 8) P-B design in Fig.
1 +++−+−− 1.1
1 is a suitable size for many experiments. If more factors need
2 −+++−+− 6.3
to be studied, a second (N 5 8) P-B design may be used. This
3 −−+++−+ 1.2
4 +−−+++− 0.8
latter procedure may involve the repeated testing of some of
5 −+−−+++ 6.0
the more important factors from the first design.
6 +−+−−++ 0.9
6.2 Ruggedness tests that have small or only moderate
7 ++−+−−+ 1.1
8 −−−−−−− 1.4 changes in the levels of the factors tend to have interactions
that are relatively small, that is, the interactions tend to be
FIG. 1 A Plackett-Burman Design for N 5 8
unimportant relative to the main effects. For such situations,
NOTICE: This standard has either been superseded and replaced by a new version or discontinued.
Contact ASTM International (www.astm.org) for the latest information.
E 1169
useful information may be obtained by investigating additional 5 8) P-B design that was used is given in Fig. 2. This
main effects rather than by investigating the numerous inter- convenient design was first suggested by F. Yates (5) and was
actions. frequently used by W. J. Youden (6) who did much of the
6.3 In general, the size of all effects in a P-B design will pioneering work in ruggedness testing. For those experienced
7-4
increase with increased separation of the high and low settings with the use of fractional factorial designs, it is a 2 design.
of the factors. It seems prudent to use only moderate separa- It has been shown (2), by a rearrangement of the rows and
tions of the high and low settings so that the measured effects columns, that this design is equivalent to the previously listed
will be approximately additive and, at the same time, reason- P-B design.
ably large relative to the measurement error. For the high and 8.2 The seven factors that were studied are listed in 8.2.1-
low settings of the factors, it is suggested that the extreme 8.2.7. The first listed level for each factor has been arbitrarily
limits that may be expected to be observed between different assigned the positive sign:
qualified laboratories be used. 8.2.1 Factor A—Temperature, 25 or 30°C,
8.2.2 Factor B—Stirring during the pH measurement: yes
7. Interpretation of Results
or no (denoted as Y or N in Table 1),
7.1 Since the main effects are expressed in the units of the
8.2.3 Factor C—Dilution (0.5 mL distilled H O/20 mL of
measurement, direct judgment can be made as to whether or
solution), yes or no,
not the change associated with the shift of the factor from a
8.2.4 Factor D—Depth of electrode immersion, 1 or 3 cm
high level to a low level is too large. Other, more quantitative
below liquid surface,
methods of judgment that analyze the variance of the measure-
8.2.5 Factor E—Addition of sodium nitrate (NaNO 5 0.67
ments are given in 7.2, 7.3, and 7.4. These quantitative
meq/20 mL solution), yes or no,
methods still only give tentative answers and follow-up or
8.2.6 Factor F—Addition of potassium chloride
confirmatory experiments are frequently needed.
(KCl 5 1.34 meq/20 mL of solution), yes or no, and
7.2 If m auxiliary measurements, all made under the same
8.2.7 Factor G—Electrode equilibration time before mea-
conditions as each other are available from other experimen-
suring the pH, 10 or 5 min.
tation, the within-laboratory measurement variability, s, can be
8.3 The seven factors are only a partial list of factors that
calculated. A t-test (with m-1 dF) can be used to judge if a main
may change the observed value of the pH. Obviously, all other
effect, such as A, is statistically significant relative to the
factors that are not listed above need to be kept constant. The
measurement variability, for example:
particular, constant levels of these other factors will result in
some specific offset in the pH measurements. In the ruggedness
effect A
t 5 (4)
m21
s test, however, this fixed offset need not be of concern since
effectA
measurement changes (the effects) that occur when the seven
Note that the m from the auxiliary measurements will not
factors (8.2.1-8.2.7) are changed is the primary interest.
generally be the same as the N of the ruggedness test. Using Eq
8.4 Results from a ruggedness test with a hydrochloric acid
3, the t-test can be calculated as follows:
(HCl) solution are given in Table 1. The complete experiment
effect A
was replicated on a second day. A different random order of
t 5 (5)
m21
2s/ N
=
measuremen
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.