ASTM E1169-21

Standard Practice for Conducting Ruggedness Tests

SIGNIFICANCE AND USE
4.1 A ruggedness test is a special application of a statistically designed experiment that makes changes in the test method variables, called factors, and then calculates the subsequent effect of those changes upon the test results. Factors are features of the test method or of the laboratory environment that are known to vary across laboratories and are subject to control by the test method.  
4.1.1 Statistical design enables more efficient and cost-effective determination of the factor effects than would be achieved if separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed for evaluating quantitative test methods.  
4.2 In ruggedness testing, the two levels (settings) for each factor are chosen to use moderate separations between the high and low settings. In general, if there is an underlying difference between the levels, then the size of effects will increase with increased separation between the high and low settings of the factors. A run is an execution of the test method under prescribed settings of each of the factors under study. A ruggedness test consists of a set of runs.  
4.3 A ruggedness test is usually conducted within a single laboratory on uniform material, so that the effects of changing only the factors are measured. The results may then be used to assist in determining the degree of control required of factors described in the test method.  
4.4 Ruggedness testing should precede an interlaboratory (round robin) study to correct any deficiencies in the test method and may also be part of the validation phase of developing a standard test method as described in Guide E1488.  
4.5 This standard discusses design and analysis of ruggedness testing in Section 5 and contains an example of a basic eight run design. Some caution must be used in interpretation of results, since interaction effects may be present. These effects are present when a factor effect changes with the ...
SCOPE
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.  
1.2 This practice restricts itself to experimental designs with two levels per factor. The designs require the simultaneous change of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.  
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations of calculation methods. The examples are not binding on products or test methods treated.  
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.5 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

General Information

• Status: Published
• Publication Date: 31-May-2021
• ICS: 19.020 - Test conditions and procedures in general
• Technical Committee: E11 - Quality and Statistics
• Drafting Committee: E11.20 - Test Method Evaluation and Quality Control

Relations

• Refers: ASTM E1488-23 - Standard Guide for Statistical Procedures to Use in Developing and Applying Test Methods
• Effective Date: 01-Nov-2023
• Refers: ASTM E2282-23 - Standard Guide for Defining the Test Result of a Test Method
• Effective Date: 01-Nov-2023
• Refers: ASTM E456-13a(2022)e1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Apr-2022
• Refers: ASTM E456-13A(2017)e1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Oct-2017
• Refers: ASTM E456-13A(2017)e3 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Oct-2017
• Refers: ASTM E1325-15 - Standard Terminology Relating to Design of Experiments
• Effective Date: 01-Oct-2015
• Refers: ASTM F2082-15 - Standard Test Method for Determination of Transformation Temperature of Nickel-Titanium Shape Memory Alloys by Bend and Free Recovery
• Effective Date: 01-May-2015
• Refers: ASTM E2282-14 - Standard Guide for Defining the Test Result of a Test Method
• Effective Date: 01-Oct-2014
• Refers: ASTM E456-13ae2 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Nov-2013
• Refers: ASTM E456-13a - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Nov-2013
• Refers: ASTM E456-13ae1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Nov-2013
• Refers: ASTM E456-13ae3 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Nov-2013
• Refers: ASTM E456-13 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 15-Aug-2013
• Refers: ASTM E2282-13 - Standard Guide for Defining the Test Result of a Test Method
• Effective Date: 01-Aug-2013
• Refers: ASTM E1488-12 - Standard Guide for Statistical Procedures to Use in Developing and Applying Test Methods
• Effective Date: 01-Aug-2012

Overview

ASTM E1169-21: Standard Practice for Conducting Ruggedness Tests defines procedures for evaluating the robustness of test methods used in laboratory environments. Developed by ASTM International, this standard provides guidance on designing and analyzing statistically valid ruggedness tests. The goal is to identify critical factors that significantly influence test results and determine how strictly these variables must be controlled to ensure test method reliability and reproducibility.

Ruggedness testing is a crucial part of test method validation, helping to uncover weaknesses in procedures before broader interlaboratory studies are conducted. By systematically varying environmental and procedural factors within controlled settings, organizations can fine-tune test methods, improve quality, and reduce variability in measurement outcomes.

Key Topics

• Purpose of Ruggedness Tests

  • Detect factors that strongly influence measurement outcomes
  • Estimate the degree of control required for each factor
  • Enhance method robustness by identifying sensitive variables

• Statistical Design

  • Utilizes two-level factorial or fractional factorial (such as Plackett-Burman) designs
  • Enables efficient and cost-effective assessment compared to evaluating factors individually
  • Only two categories (low/high or equivalent) per factor are considered

• Test Execution

  • Runs are conducted by systematically varying factors according to the design matrix
  • Typically performed in a single laboratory using uniform material
  • Focuses solely on the effect of controlled variables

• Data Analysis and Interpretation

  • Main effects are calculated by comparing average results at different factor levels
  • Half-normal plots are used to visually identify significant effects
  • Statistical significance is assessed to determine if method modifications are necessary
  • Awareness of interaction effects is vital, as simple designs may confound them (aliasing)

• Validation and Method Development

  • Ruggedness testing precedes interlaboratory studies (round robin) for comprehensive method validation
  • Supports compliance with international best practices in test method development

Applications

• Test Method Development

  • Validation of new or modified laboratory test protocols in industries like materials testing, pharmaceuticals, and environmental monitoring
  • Early detection of factors that might cause test failures or inaccuracies

• Quality Assurance

  • Ensuring consistency and comparability of analytical results across different laboratories or over time
  • Identifying the variables that require strict control to meet regulatory or customer requirements

• Standards Compliance

  • Satisfying requirements for method validation specified in other ASTM, ISO, and regulatory guidelines
  • Supporting evidence for audit or regulatory review processes by demonstrating method robustness

• Troubleshooting and Continuous Improvement

  • Diagnosing sources of unexpected variability in measurement results
  • Facilitating root cause analysis and ongoing process refinement

Related Standards

For comprehensive method development, ASTM E1169-21 should be used in conjunction with other key ASTM standards:

• ASTM E1488: Guide for Statistical Procedures to Use in Developing and Applying Test Methods
• ASTM E456: Terminology Relating to Quality and Statistics
• ASTM E1325: Terminology Relating to Design of Experiments
• ASTM E2282: Guide for Defining the Test Result of a Test Method
• ASTM F2082: Test Method for Determination of Transformation Temperature of Nickel-Titanium Shape Memory Alloys by Bend and Free Recovery

Practical Value

Implementing ASTM E1169-21 allows laboratories and organizations to develop reliable, reproducible, and robust test methods. By identifying the factors that matter most, the standard reduces experiment costs, saves time, and ensures reliable data for decision-making. Proper ruggedness testing forms the foundation for quality analytical practices and test method standardization, contributing to improved product development, regulatory compliance, and industry best practices.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E1169 − 21 An American National Standard
Standard Practice for
Conducting Ruggedness Tests
This standard is issued under the fixed designation E1169; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope E2282Guide for Defining the Test Result of a Test Method
F2082Test Method for Determination of Transformation
1.1 This practice covers conducting ruggedness tests. The
Temperature of Nickel-Titanium Shape Memory Alloys
purpose of a ruggedness test is to identify those factors that
by Bend and Free Recovery
stronglyinfluencethemeasurementsprovidedbyaspecifictest
method and to estimate how closely those factors need to be
3. Terminology
controlled.
3.1 Definitions—Unlessotherwisenotedinthisstandard,all
1.2 Thispracticerestrictsitselftoexperimentaldesignswith
terms relating to quality and statistics are defined in Terminol-
two levels per factor. The designs require the simultaneous
ogy E456.
change of the levels of all of the factors, thus permitting the
3.1.1 factor, n—independent variable in an experimental
determination of the effects of each of the factors on the
design. E1325
measured results.
3.1.1.1 Discussion—For experimental purposes, factors
1.3 The system of units for this practice is not specified.
must be temporarily controllable. In a ruggedness test, a factor
Dimensional quantities in the practice are presented only as
is a test variable that may affect either the result obtained from
illustrations of calculation methods. The examples are not
the use of the test method or the variability of the result.
binding on products or test methods treated.
3.1.2 fractional factorial design, n—a factorial experiment
1.4 This standard does not purport to address all of the
in which only an adequately chosen fraction of the treatments
safety concerns, if any, associated with its use. It is the
required for the complete factorial experiment is selected to be
responsibility of the user of this standard to establish appro-
run. E1325
priate safety, health, and environmental practices and deter-
mine the applicability of regulatory limitations prior to use.
3.1.3 interaction, n—differences in responses to a factor
1.5 This international standard was developed in accor-
among levels (versions) of other factors in the experiment.
dance with internationally recognized principles on standard-
E1325
ization established in the Decision on Principles for the
3.1.3.1 Discussion—Interaction is the condition where a
Development of International Standards, Guides and Recom-
factor effect changes with the level of other factors in the
mendations issued by the World Trade Organization Technical
experiment design.
Barriers to Trade (TBT) Committee.
3.1.4 level (of a factor), n—a given value, a specification of
2. Referenced Documents
procedure or a specific setting of a factor. E1325
2.1 ASTM Standards:
3.1.5 main effect, average effect, n—a term describing a
E456Terminology Relating to Quality and Statistics
measure for the comparison of the responses at each level
E1325Terminology Relating to Design of Experiments
(version)ofafactoraveragedoveralllevels(versions)ofother
E1488GuideforStatisticalProcedurestoUseinDeveloping
factors in the experiment. E1325
and Applying Test Methods
3.1.5.1 Discussion—This is also known as a first-order
effect. In a ruggedness test, the main effect is the change in the
test result due to a change in the level of a factor. This is the
ThispracticeisunderthejurisdictionofASTMCommitteeE11onQualityand
Statistics and is the direct responsibility of Subcommittee E11.20 on Test Method
difference of the average result at the high level of the factor
Evaluation and Quality Control.
minus the average result at the low level. There are only two
CurrenteditionapprovedJune1,2021.PublishedJuly2021.Originallyapproved
levels in the ruggedness tests considered here.
in 1987. Last previous edition approved in 2020 as E1169–20. DOI: 10.1520/
E1169-21.
3.1.6 Plackett-Burman designs, n—a set of screening de-
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
signs using orthogonal arrays that permit evaluation of the
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM
linear effects of up to n = t – 1 factors in a study of t treatment
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website. combinations. E1325
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E1169 − 21
3.1.7 ruggedness, n—insensitivity of a test method to de- environmentthatareknowntovaryacrosslaboratoriesandare
partures from specified test or environmental conditions. subject to control by the test method.
3.1.7.1 Discussion—An evaluation of the “ruggedness” of a
4.1.1 Statistical design enables more efficient and cost-
test method or an empirical model derived from an experiment effective determination of the factor effects than would be
isusefulindeterminingwhethertheresultsordecisionswillbe
achieved if separate experiments were carried out for each
relativelyinvariantoversomerangeofenvironmentalvariabil- factor. The proposed designs are easy to use in developing the
ity under which the test method or the model is likely to be
information needed for evaluating quantitative test methods.
applied.
4.2 In ruggedness testing, the two levels (settings) for each
3.1.8 ruggedness test, n—a planned experiment in which
factorarechosentousemoderateseparationsbetweenthehigh
environmental factors or test conditions are deliberately varied
andlowsettings.Ingeneral,ifthereisanunderlyingdifference
in order to evaluate the effects of such variation.
between the levels, then the size of effects will increase with
3.1.8.1 Discussion—Since there usually are many environ-
increased separation between the high and low settings of the
mental factors that might be considered in a ruggedness test, it
factors. A run is an execution of the test method under
is customary to use a “screening” type of experiment design
prescribed settings of each of the factors under study. A
which concentrates on examining many first order effects. The
ruggedness test consists of a set of runs.
validityoftheestimatesdependsontheassumptionthatsecond
4.3 A ruggedness test is usually conducted within a single
order effects such as interactions and curvature are relatively
laboratory on uniform material, so that the effects of changing
negligible.Ofteninevaluatingtheruggednessofatestmethod,
only the factors are measured.The results may then be used to
if there is an indication that the results of a test method are
assist in determining the degree of control required of factors
highly dependent on the levels of the environmental factors,
described in the test method.
there is a sufficient indication that certain levels of environ-
mentalfactorsmustbeincludedinthespecificationsforthetest
4.4 Ruggedness testing should precede an interlaboratory
method, or even that the test method itself will need further
(round robin) study to correct any deficiencies in the test
revision. This evaluation may include extra runs in a second
method and may also be part of the validation phase of
experiment.
developing a standard test method as described in Guide
E1488.
3.1.9 screening design, n—a balanced design, requiring
relatively minimal amount of experimentation, to evaluate the
4.5 This standard discusses design and analysis of rugged-
lower order effects of a relatively large number of factors in
ness testing in Section 5 and contains an example of a basic
terms of contributions to variability or in terms of estimates of
eight run design. Some caution must be used in interpretation
parameters for a model. E1325
of results, since interaction effects may be present. These
effects are present when a factor effect changes with the level
3.1.10 test result, n—the value of a characteristic obtained
of other factors in the experimental design. If it is thought that
by carrying out a specified test method. E2282
there may be interaction between variables then additional
3.1.11 test unit, n—thetotalquantityofmaterial(containing
testing of the basic design is necessary. This is discussed in
one or more specimens) needed to obtain a test result as
Section 6. In addition, Annex A3 presents estimates of preci-
specified in the test method. E2282
sion of factor effects when run settings are replicated. An
3.2 Definitions of Terms Specific to This Standard:
example of a twelve run design is shown in Appendix X1.
3.2.1 foldover, n—test runs, added to a two-level fractional
Annex A1 and Annex A2 provide supplemental information.
factorial experiment, generated by duplicating the original
design by switching levels of one or more factors in all runs,
5. Basic Ruggedness Test Design and Analysis
for the purpose of separating estimates of main effects from
5.1 Design—A series of fractional factorial designs are
two factor interactions.
recommendedforusewithruggednesstestsfordeterminingthe
3.2.1.1 Discussion—The most useful type of foldover is
factor effects on the test results. All designs considered here
with signs of all factors switched. The foldover runs are
have only two settings (levels) for each factor, and are known
combined with the initial test results. The combination allows
as Plackett-Burman (PB) designs (1). These designs occur in
main effects to be separated from interactions of other factors
multiples of four runs, such as 4, 8, 12, etc., and are listed in
that are aliased in the original design.
Annex A1, Each run conducts the test method at designated
3.2.2 two-factor interaction effect, 2fi, n—estimate of the
levelsofthefactorstoproduceatestresultastherunresponse.
condition where a factor effect changes with the level of
5.1.1 Select k factors to investigate. Choose a PB design
another factor in the experiment design.
with at least k+1 runs. Assign each factor to a column in the
designtable.Theunassignedcolumnsinthedesignaredenoted
4. Significance and Use
as “dummy” factors, and these may be used to estimate the
experimental error (see 5.2.3.2).
4.1 A ruggedness test is a special application of a statisti-
cally designed experiment that makes changes in the test
method variables, called factors, and then calculates the
subsequent effect of those changes upon the test results.
Theboldfacenumbersinparenthesesrefertothelistofreferencesattheendof
Factors are features of the test method or of the laboratory this standard.
E1169 − 21
5.1.2 Choose the factor levels for each factor such that the 5.2 Analysis—The analysis of the experimental results con-
measured effects will be reasonably large relative to measure- sists of (1) calculating the main effects for each of the factors,
menterror.Itissuggestedthatthehighandlowlevelsbesetat includingdummyfactors,ifany,(2)creatingahalf-normalplot
the extreme limits that could be expected to exist between to exhibit the magnitudes of the factor main effects, (3)
different qualifying laboratories. assessingthestatisticalsignificanceofeachfactor’smaineffect
5.1.3 Factorlevelsmaybeeithernumericalorcategorical.If if an estimate of experimental error is available.
categorical, only two categories are permitted in the design. If
5.2.1 Main Effect Estimation:
the lower level for a numerical factor is zero, then the factor is
5.2.1.1 For each factor column calculate the average of the
essentially categorical (that is, the factor is either present or
test results corresponding to the + factor level and enter the
not).
result in theAve+ row in the column. Calculate the average of
5.1.4 As an example for this section, Table 1 shows the PB
thetestresultscorrespondingtothe–factorlevelandenterthe
eight run design for up to seven factors, with factors denoted
result in the Ave– row in the column.
by the lettersA–G. Each row lists the factor levels for each of
5.2.1.2 Calculate the Main Effect = (Ave+ entry) minus
the eight runs as indicated by either (–1) or (1) for low or high
(Ave– entry).
levels, respectively. For factors with non-numerical scales
5.2.2 Half-Normal Plot—A half-normal plot is used to
(categorical), the designation “low” or “high” is arbitrary.
graphically identify active effects.
5.1.5 The design provides equal numbers of low and high
5.2.2.1 Constructahalf-normalplotbyplottingtheabsolute
level runs for every factor. In other words, the designs are
values of main effects on the X-axis, in order from smallest to
balanced. Also, for any factor, while it is at its high level, all
largest, against the half-normal plotting values given in Annex
other factors will be run at equal numbers of high and low
A2 on the Y-axis. Effects for all columns in the design,
levels;similarly,whileitisatitslowlevel,allotherfactorswill
including columns not used to assign levels to any real
be run at equal numbers of high and low levels. In the
experiment factor, are plotted. The half-normal plotting values
terminology used by statisticians, the design is orthogonal.
do not depend on the effect values. They depend only on the
5.1.6 Thedifferencebetweentheaverageresponseofrunsat
half-normal distribution and the number of effects plotted.
thehighlevelandtheaverageresponseofrunsatthelowlevel
5.2.2.2 Ifnoneofthefactorshaveameasurablemaineffect,
of a factor is the estimated “main effect” of that factor. This
the plotted points will form a straight line. Factors having true
estimate is then used to quantify the factor’s effect on the test
effects will lie to the right of the reference line formed by the
result.
smaller effects.
5.1.7 Run Order—The sequence of runs in Table 1 is not
intended to be the actual sequence for carrying out the 5.2.2.3 Ifanestimateofthestandarddeviationoftestresults
(s ), which may be either repeatability or laboratory precision,
experiments. The order in which the runs of a ruggedness
tr
experiment are carried out should be randomized to reduce the is available, a reference line in the half-normal plot can be
probability of encountering any potential effects of unknown, @ = #
provided with slope1⁄s 51⁄ s 3 ~4⁄ N! . Effects that fall
effect tr
time-related factors. The run order is to be listed in the second
farthesttotherightofthelinemaybeconsideredactiveeffects.
column of Table 1 for use by the experimenter. Alternatively,
5.2.2.4 If an estimate of error is not available, a reference
optimum run orders to control the number of required factor
line may be drawn by eye for the purpose of identifying
changes and the effect of linear time trends have been derived
potentially significant effects. Select points with the smallest
(2). In some cases, it is not possible to change all factors in a
effects that appear to fall on a straight line from the origin.
completely random order. It is best if this limitation is
Draw a line starting at the origin and passing close to the
understood before the start of the experiment. A statistician
selected points.
may be contacted for methods to deal with such situations.
5.1.8 The test results are entered in the last column of Table
NOTE 1—The slope of the line fit by eye does not provide a valid
1 for data analysis. estimate of error when measurable effects appear to be present.
TABLE 1 PB Eight Run Design for Up to Seven Factors
NOTE 1—For four factors, use ColumnsA, B, C, and E; for five factors, use ColumnsA, B, C, D, and F; for six factors, use ColumnsA, B, C, D, F,
andG.
PB Order Run Order A B C D E F G Test Result
1 111 –1 1 –1 –1
2 –1111 –1 1 –1
3–1–1111–11
4 1 –1 –1 1 1 1 –1
5 –1 1 –1 –1 1 1 1
61–11–1–111
7 1 1 –1 1 –1 –1 1
8 –1–1–1–1–1–1 –1
Ave+ — — — — — — — —
Ave– — — — — — — — —
Effect — — — — — — — —
E1169 − 21
5.2.3 Statistical Significance Testing—If an estimate of pre- and their levels are shown in Table 2. Factors C, D, and G are
cision is available or can be derived from the experiment (see numerical, and the rest are categorical. There are no dummy
AnnexA3), statistical tests of factor effects can be determined factors, so the design is said to be saturated (all columns
using the Student’s t-test. The t-test statistic for a factor is the assigned to factors).
main effect divided by its standard error s , which is the same 5.3.2 The data and calculated main effects for the initial
e
for all factors in a balanced and orthogonal design. If the design are shown in Table 3. The results are recorded as 1000
t-value is greater than the t-value corresponding to the 0.05 pH.
significance level, the factor is statistically significant at the 5.3.3 In Table 3, theAve+ value for factorAis the average
95% confidence level. of the four measurements at the “1” value for A (dilution of
5.2.3.1 If fewer factors are used with the design than the water): 3015, 2964, 2949 and 3055, the average of which is
maximum number, then the “dummy effects” estimated for the 2995.75. The Ave– value is the average of the four measure-
unusedcolumnsdifferfromzeroonlyasaresultofexperimen- ments at the “–1” value for A (no dilution of water): 3006,
talerror(orinteractionsofotherfactors).Therootmeansquare 2999,3049and2904,theaverageofwhichis2989.5.Themain
of unused effects is an estimate of the standard error of an effect is the difference of these values 2995.75 – 2989.5 = 6.3.
effect having degrees of freedom equal to the number of Theothereffectestimatesarecalculatedanalogously:B=77.3,
unused effects averaged (1). C = –0.8, D = 26.8, E = 28.3, F = –1.3, G = 40.8.
5.2.3.2 In this case, the line described in 5.2.2.3 has slope 5.3.4 Half-Normal Plot—The half-normal plotting values
1/2
1 ⁄(MSE) , where MSE denotes the mean square error of the are shown in Table 4.As was suggested in 5.2.2.4, a reference
dummy effects (see 5.1.1). For instance, if there are three line which passes close to the three smallest values is added to
dummy effects, e1, e2, and e3, then MSE = the figure. From the half-normal plot in Fig. 1, we see that
2 2 2
(e1 + e2 + e3)⁄3. factors B, G, E, and D appear to be active.
5.2.4 Ruggedness Test Conclusions: 5.3.5 Although the data discussed here give evidence of
5.2.4.1 If no effects are identified as statistically significant active effects, that will not always happen. When no effects
and practically important, and if the experimenter is satisfied appear active, the method shows no evidence of lack of
with the way that the experiment was carried out and with its ruggedness.Whenthereareactiveeffects,itmaybeofvalueto
statistical power, then there is reason to think that the method do further experimentation to find active two-factor
is rugged with regard to the factors tested. interactions, as discussed in Section 6.
5.2.4.2 If some effects are identified as statistically signifi-
cantandpracticallyimportant,thenthemethodmayhavetobe 6. Separating Main Effects and Two-Factor Interactions
modified, or specifications may need to be added for the range with Added Foldover Runs
of acceptable values of the identified factors. In cases where
6.1 Interactions—If the effect of one factor depends on the
the factor effects may be statistically significant but not
level of another factor, then these two factors interact. As
practically important the method can still be classified as
shown in Section 5, a main effect for a factor is estimated by
“rugged.”
the difference between the mean measurement of the four high
5.2.5 Supplemental Additions to the Basic Ruggedness Test
level measurements and the mean of the four low level
Design:
measurements. By contrast, the two-factor interaction between
5.2.5.1 The basic design allows only for the estimation of
C and D is estimated as follows. For the high level of factor C
main effects. When there is uncertainty whether the factor
compute the difference (high level D mean – low level D
effectschangewiththelevelsofotherfactorsintheexperiment
mean), where each mean is the average of two measurements.
(3, 4), the main effects may be separated from the interaction
Calculate the corresponding difference at the low level. Half
effectsbyconductingadditionalruns,asdiscussedinSection6.
the difference of these two differences is the interaction
5.2.5.2 Each of the runs may be replicated to obtain an
betweenfactorsCandD,inthesensethatifthefactorDeffect
estimateofexperimentalvariabilityinadditiontothatsupplied
does not vary by levels of factor C, the two factor interaction
bydummyfactors,andthismaybeconductedinthreedifferent
calculatedaboveshouldbecloseto0.Itturnsoutthattheeight
ways, as discussed in Annex A3.
signsforColumnCofTable1,multipliedbythecorresponding
5.3 Example: eight signs in Column D, give a column of signs that specifies
5.3.1 The example discussed here is part of a series of this same CD interaction if we take the difference of the
experiments that studied the effects of factors that influence average of the measurements that correspond to “1” and the
determination of pH in dilute acid solutions (3, 4). The factors average of the measurements that correspond to “–1”. In
TABLE 2 Example: Factors That Influence Determination of pH in Dilute Acid Solutions
Factor No. Variable Units Level 1 (–) Level 2 (+)
A Dilution with water yes or no No yes
B Addition of potassium chloride yes or no No yes
C Equilibration time minutes 5 10
D Depth of electrode immersion cm 1 3
E Addition of sodium nitrate yes or no No yes
F Stirring yes or no No yes
G Temperature °C 2 4
E1169 − 21
TABLE 3 Results and Effects for Initial Design
PB Test
AB C D E F G
Order Result
1 1 1 1 –1 1 –1 –1 3015
2 –1 1 1 1 –1 1 –1 3006
3 –1 –1 1 1 1 –1 1 2999
4 1 –1 –1 1 1 1 –1 2964
5 –1 1 –1 –1 1 1 1 3049
6 1 –1 1 –1 –1 1 1 2949
7 1 1 –1 1 –1 –1 1 3055
8 –1 –1 –1 –1 –1 –1 –1 2904
Ave+ 2995.8 3031.3 2992.3 3006.0 3006.8 2992.0 3013.0
Ave– 2989.5 2954.0 2993.0 2979.3 2978.5 2993.3 2972.3
Main Effect 6.3 77.3 –0.8 26.8 28.3 –1.3 40.8
TABLE 4 Estimated Effects and Half-Normal Plotting Values
Half-Normal
Effect Order, e Effect Estimated Effect
Plotting Values
7 B 77.3 1.8
6 G 40.8 1.24
5 E 28.3 0.92
4 D 26.8 0.67
3 A 6.3 0.46
2 F 1.3 0.27
1 C 0.8 0.09
FIG. 1 Half-Normal Plot of pH Data
addition, it turns out that the negatives of these eight values is interaction, written as –CD. There is no way to know whether
the same as the columnAvalues. Thus, when we calculate the the main effect for A is really estimating that factor or the
factor A effect we are also calculating the negative of the CD negative of the interaction between factors C and D.
E1169 − 21
TABLE 5 Factorial Effect Aliases for Design in Table 1
confounded interactions are estimated by taking half the
[A]=A– BF–CD–EG difference of the main effect estimates.
[B]=B–AF–CG–DE
6.4 Half-Normal Plot—Using data from the initial runs and
[C]=C–AD–BG–EF
[D]=D–AC–BE–FG
the foldover together, the effects are ordered by absolute value
[E]=E–AG–BD–CF
andshownwiththeassociatedhalf-normalplotvaluesinTable
[F]=F–AB–CE–DG
8 and plotted in Fig. 2. The suffix –I, added to a factor label,
[G]=G–AE–BC–DF
indicates the two factor interactions that are confounded with
the factor. The nine smallest estimates appear to lie approxi-
mately on a straight line, drawn in Fig. 2, following 5.2.2.4.
6.1.1 Thus, the complication of the fractional factorial
designs presented in Section 5 is that each main effect is From the distribution of points in the plot, factors B, G, E, and
D–Iappeartobeactive.WhetherfactorG–Iisactiveisunclear.
confounded (aliased) with a group of two-factor interactions,
as shown in Table 5. Note that factor A is confounded with
6.5 In Table 5, it is shown that InteractionsAF, CG, and DE
threetwo-wayinteractions,onebeing–CD,whichisdiscussed
are confounded with factor B. Thus, there is no way to know
above. The other confounded interactions are –BF and –EG.
whether the apparent significance of factor B is due to a
Factorsaresaidtobe“aliased”whentheircolumnsofsignsare
confounded interaction.As a general rule, factors interact only
the negatives or positives of each other.
when they have large main effects in their own right. Hence,
6.2 Design—To separate factor main effects from groups of AF and CG are unlikely to be important, but a DE interaction
could be contributing to the estimated B effect. Similarly,AC,
two-factor interactions, the PB design is augmented with eight
additional runs called a foldover. A set of foldover runs is BE, and FG are confounded with D; a BE interaction could be
contributing.
generated from the original design by changing all the 1’s to
–1’s and the –1’s to 1’s.Thus, in the foldover, for each run, for
6.6 When factors are separated from confounded
each factor level of the initial design of Table 3, the opposite
interactions, it appears that factor D is not active, but the
level in each factor is used.As will be seen below, combining
apparent significance of D in the initial portion of the experi-
the two sets of runs will allow us to estimate the main effects
ment was due to confounded interactions. The most likely
without confounding from the two-factor interactions. The set
cause of the large D–I two-factor interaction is the BE
of foldover runs for the eight run PB design is shown in Table
interaction, since the main effects B and E are the largest,
6, together with the test results and calculated main effects for
though only additional experimentation can confirm this.
these eight runs.
7. Keywords
6.3 Analysis—To combine the results of original design and
foldoverinTable7,themaineffectsareestimatedbyaveraging 7.1 foldover; fractional factorial design; half-normal plot;
the main effect estimates from the two sets.The corresponding Plackett-Burman; ruggednessscreening design
E1169 − 21
TABLE 6 Results and Effects for Foldover Factor—Settings Are at the Opposite Level to the First Set (Table 3)
Test
PB Order A B C D E F G
Result
1 –1 –1 –1 1 –1 1 1 2931
2 1 –1 –1 –1 1 –1 1 2978
3 1 1 –1 –1 –1 1 –1 2967
4 –1 1 1 –1 –1 –1 1 3030
5 1 –1 1 1 –1 –1 –1 2874
6 –1 1 –1 1 1 –1 –1 2979
7 –1 –1 1 –1 1 1 –1 2911
8 1111111 3040
Ave+ 2964.8 3004.0 2963.8 2956.0 2977.0 2962.3 2994.8
Ave– 2962.8 2923.5 2963.8 2971.5 2950.5 2965.3 2932.8
Main Effect 2.0 80.5 0.0 –15.5 26.5 –3.0 62.0
TABLE 7 Calculation of Estimated Effects Using Data from
Table 3 and Table 6
Foldover
Factor Table 3 Average
(Table 6)
A 6.3 2.0 4.1
B 77.3 80.5 78.9
C –0.8 0.0 –0.4
D 26.8 –15.5 5.6
E 28.3 26.5 27.4
F –1.3 –3.0 –2.1
G 40.8 62.0 51.4
⁄2 difference
A–I = –BF – CD – EG 6.3 2.0 –2.1
B–I = –AF – CG – DE 77.3 80.5 1.6
C–I = –AD – BG – EF –0.8 0.0 0.38
D–I = –AC – BE – FG 26.8 –15.5 –21.1
E–I = –AG – BD – CF 28.3 26.5 –0.88
F–I = –AB – CE – DG –1.3 –3.0 –0.88
G–I = –AE – BC – DF 40.8 62.0 10.6
TABLE 8 Ordered Effects and Half-Normal Plotting Positions
Half-Normal
Factor Effect Abs (Effect)
Plotting Value
B 78.9 78.9 2.100
G 51.4 51.4 1.611
E 27.4 27.4 1.345
D–I –21.1 21.1 1.150
G–I 10.6 10.6 0.992
D 5.6 5.6 0.854
A 4.1 4.1 0.732
A–I –2.1 2.1 0.619
F –2.1 2.1 0.514
B–I 1.6 1.6 0.414
F–I –0.88 0.88 0.319
E–I –0.88 0.88 0.226
C–I 0.38 0.38 0.135
C –0.38 0.38 0.045
E1169 − 21
FIG. 2 Half-Normal Plot, Foldover pH Experiment
ANNEXES
(Mandatory Information)
A1. ADDITIONAL PLACKETT-BURMAN DESIGNS
A1.1 Plackett-Burmandesigns (1)areavailablefor Nvalues A1.2 For any selected N value, the corresponding set of
thatareintegermultiplesoffour.Thefollowingisamethodfor N–1 (1) and (–1) signs is written down as the first row of the
constructing the designs for N = 4, 8, 12, 16, 20, and 24. The
design. The second row of the design is obtained by copying
first row of each of these designs is given below for the
the first row after shifting it one place to the right and putting
associated N value. Each row specifies which of the N–1
the las
...

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E1169 − 20 E1169 − 21 An American National Standard
Standard Practice for
Conducting Ruggedness Tests
This standard is issued under the fixed designation E1169; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope
1.1 This practice covers conducting ruggedness tests. The purpose of a ruggedness test is to identify those factors that strongly
influence the measurements provided by a specific test method and to estimate how closely those factors need to be controlled.
1.2 This practice restricts itself to experimental designs with two levels per factor. The designs require the simultaneous change
of the levels of all of the factors, thus permitting the determination of the effects of each of the factors on the measured results.
1.3 The system of units for this practice is not specified. Dimensional quantities in the practice are presented only as illustrations
of calculation methods. The examples are not binding on products or test methods treated.
1.4 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.5 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E456 Terminology Relating to Quality and Statistics
E1325 Terminology Relating to Design of Experiments
E1488 Guide for Statistical Procedures to Use in Developing and Applying Test Methods
E2282 Guide for Defining the Test Result of a Test Method
F2082 Test Method for Determination of Transformation Temperature of Nickel-Titanium Shape Memory Alloys by Bend and
Free Recovery
3. Terminology
3.1 Definitions—The terminology Unless otherwise noted in this standard, all terms relating to quality and statistics are defined
in Terminology E456 applies to this practice unless modified herein.
3.1.1 factor, n—independent variable in an experimental design. E1325
This practice is under the jurisdiction of ASTM Committee E11 on Quality and Statistics and is the direct responsibility of Subcommittee E11.20 on Test Method
Evaluation and Quality Control.
Current edition approved April 1, 2020June 1, 2021. Published May 2020July 2021. Originally approved in 1987. Last previous edition approved in 20182020 as
E1169 – 18.E1169 – 20. DOI: 10.1520/E1169-20.10.1520/E1169-21.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM Standards
volume information, refer to the standard’sstandard’s Document Summary page on the ASTM website.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E1169 − 21
3.1.1.1 Discussion—
For experimental purposes, factors must be temporarily controllable. In a ruggedness test, a factor is a test variable that may affect
either the result obtained from the use of the test method or the variability of the result.
3.1.2 fractional factorial design, n—a factorial experiment in which only an adequately chosen fraction of the treatments required
for the complete factorial experiment is selected to be run. E1325
3.1.3 interaction, n—differences in responses to a factor among levels (versions) of other factors in the experiment. E1325
3.1.3.1 Discussion—
Interaction is the condition where a factor effect changes with the level of other factors in the experiment design.
3.1.4 level (of a factor), n—a given value, a specification of procedure or a specific setting of a factor. E1325
3.1.5 main effect, average effect, n—a term describing a measure for the comparison of the responses at each level (version) of
a factor averaged over all levels (versions) of other factors in the experiment. E1325
3.1.5.1 Discussion—
This is also known as a first-order effect. In a ruggedness test, the main effect is the change in the test result due to a change in
the level of a factor. This is the difference of the average result at the high level of the factor minus the average result at the low
level. There are only two levels in the ruggedness tests considered here.
3.1.6 Plackett-Burman designs, n—a set of screening designs using orthogonal arrays that permit evaluation of the linear effects
of up to n = t – 1 factors in a study of t treatment combinations. E1325
3.1.7 ruggedness, n—insensitivity of a test method to departures from specified test or environmental conditions.
3.1.7.1 Discussion—
An evaluation of the “ruggedness” of a test method or an empirical model derived from an experiment is useful in determining
whether the results or decisions will be relatively invariant over some range of environmental variability under which the test
method or the model is likely to be applied.
3.1.8 ruggedness test, n—a planned experiment in which environmental factors or test conditions are deliberately varied in order
to evaluate the effects of such variation.
3.1.8.1 Discussion—
Since there usually are many environmental factors that might be considered in a ruggedness test, it is customary to use a
“screening” type of experiment design which concentrates on examining many first order effects. The validity of the estimates
depends on the assumption that second order effects such as interactions and curvature are relatively negligible. Often in evaluating
the ruggedness of a test method, if there is an indication that the results of a test method are highly dependent on the levels of the
environmental factors, there is a sufficient indication that certain levels of environmental factors must be included in the
specifications for the test method, or even that the test method itself will need further revision. This evaluation may include extra
runs in a second experiment.
3.1.9 screening design, n—a balanced design, requiring relatively minimal amount of experimentation, to evaluate the lower order
effects of a relatively large number of factors in terms of contributions to variability or in terms of estimates of parameters for a
model. E1325
3.1.10 test result, n—the value of a characteristic obtained by carrying out a specified test method. E2282
3.1.11 test unit, n—the total quantity of material (containing one or more specimens) needed to obtain a test result as specified in
the test method. E2282
3.2 Definitions of Terms Specific to This Standard:
3.2.1 factor, n—test variable that may affect either the result obtained from the use of a test method or the variability of that result.
3.2.1.1 Discussion—
For experimental purposes, factors must be temporarily controllable.
E1169 − 21
3.2.1 foldover, n—test runs, added to a two-level fractional factorial experiment, generated by duplicating the original design by
switching levels of one or more factors in all runs, for the purpose of separating estimates of main effects from two factor
interactions.
3.2.1.1 Discussion—
The most useful type of foldover is with signs of all factors switched. The foldover runs are combined with the initial test results.
The combination allows main effects to be separated from interactions of other factors that are aliased in the original design.
3.2.3 interaction, n—condition where a factor effect changes with the level of other factors in the experiment design.
3.2.4 main effect, n—in a ruggedness test, the change in the test result due to a change in the level of a factor.
3.2.4.1 Discussion—
This is also known as a first-order effect.
3.2.2 two-factor interaction effect, 2fi, n—estimate of the condition where a factor effect changes with the level of another factor
in the experiment design.
4. Significance and Use
4.1 A ruggedness test is a special application of a statistically designed experiment that makes changes in the test method
variables, called factors, and then calculates the subsequent effect of those changes upon the test results. Factors are features of
the test method or of the laboratory environment that are known to vary across laboratories and are subject to control by the test
method.
4.1.1 Statistical design enables more efficient and cost-effective determination of the factor effects than would be achieved if
separate experiments were carried out for each factor. The proposed designs are easy to use in developing the information needed
for evaluating quantitative test methods.
4.2 In ruggedness testing, the two levels (settings) for each factor are chosen to use moderate separations between the high and
low settings. In general, if there is an underlying difference between the levels, then the size of effects will increase with increased
separation between the high and low settings of the factors. A run is an execution of the test method under prescribed settings of
each of the factors under study. A ruggedness test consists of a set of runs.
4.3 A ruggedness test is usually conducted within a single laboratory on uniform material, so that the effects of changing only the
factors are measured. The results may then be used to assist in determining the degree of control required of factors described in
the test method.
4.4 Ruggedness testing should precede an interlaboratory (round robin) study to correct any deficiencies in the test method and
may also be part of the validation phase of developing a standard test method as described in Guide E1488.
4.5 This standard discusses design and analysis of ruggedness testing in Section 5 and contains an example of a basic eight run
design. Some caution must be used in interpretation of results, since interaction effects may be present. These effects are present
when a factor effect changes with the level of other factors in the experimental design. If it is thought that there may be interaction
between variables then additional testing of the basic design is necessary. This is discussed in Section 6. In addition, Annex A3
presents estimates of precision of factor effects when run settings are replicated. An example of a twelve run design is shown in
Appendix X1. Annex A1 and Annex A2 provide supplemental information.
5. Basic Ruggedness Test Design and Analysis
5.1 Design—A series of fractional factorial designs are recommended for use with ruggedness tests for determining the factor
effects on the test results. All designs considered here have only two settings (levels) for each factor, and are known as
Plackett-Burman (PB) designs (1, 2). These designs occur in multiples of four runs, such as 4, 8, 12, etc., and are listed in Annex
A1, Each run conducts the test method at designated levels of the factors to produce a test result as the run response.
The boldface numbers in parentheses refer to the list of references at the end of this standard.
E1169 − 21
5.1.1 Select k factors to investigate. Choose a PB design with at least k+1 runs. Assign each factor to a column in the design table.
The unassigned columns in the design are denoted as “dummy” factors, and these may be used to estimate the experimental error
(see 5.2.3.2).
5.1.2 Choose the factor levels for each factor such that the measured effects will be reasonably large relative to measurement error.
It is suggested that the high and low levels be set at the extreme limits that could be expected to exist between different qualifying
laboratories.
5.1.3 Factor levels may be either numerical or categorical. If categorical, only two categories are permitted in the design. If the
lower level for a numerical factor is zero, then the factor is essentially categorical (that is, the factor is either present or not).
5.1.4 As an example for this section, Table 1 shows the PB eight run design for up to seven factors, with factors denoted by the
letters A–G. Each row lists the factor levels for each of the eight runs as indicated by either (–1) or (1) for low or high levels,
respectively. For factors with non-numerical scales (categorical), the designation “low” or “high” is arbitrary.
5.1.5 The design provides equal numbers of low and high level runs for every factor. In other words, the designs are balanced.
Also, for any factor, while it is at its high level, all other factors will be run at equal numbers of high and low levels; similarly,
while it is at its low level, all other factors will be run at equal numbers of high and low levels. In the terminology used by
statisticians, the design is orthogonal.
5.1.6 The difference between the average response of runs at the high level and the average response of runs at the low level of
a factor is the estimated “main effect” of that factor. This estimate is then used to quantify the factor’s effect on the test result.
5.1.7 Run Order—The sequence of runs in Table 1 is not intended to be the actual sequence for carrying out the experiments. The
order in which the runs of a ruggedness experiment are carried out should be randomized to reduce the probability of encountering
any potential effects of unknown, time-related factors. The run order is to be listed in the second column of Table 1 for use by the
experimenter. Alternatively, optimum run orders to control the number of required factor changes and the effect of linear time
trends have been derived (32). In some cases, it is not possible to change all factors in a completely random order. It is best if this
limitation is understood before the start of the experiment. A statistician may be contacted for methods to deal with such situations.
5.1.8 The test results are entered in the last column of Table 1 for data analysis.
5.2 Analysis—The analysis of the experimental results consists of (1) calculating the main effects for each of the factors, including
dummy factors, if any, (2) creating a half-normal plot to exhibit the magnitudes of the factor main effects, (3) assessing the
statistical significance of each factor’s main effect if an estimate of experimental error is available.
TABLE 1 PB Eight Run Design for Up to Seven Factors
NOTE 1—For four factors, use Columns A, B, C, and E; for five factors, use Columns A, B, C, D, and F; for six factors, use Columns A, B, C, D, F,
and G.
PB Order Run Order A B C D E F G Test Result
1 1 1 1 –1 1 –1 –1
1 1 1 1 –1 1 –1 –1
2 –1 1 1 1 –1 1 –1
2 –1 1 1 1 –1 1 –1
3 –1 –1 1 1 1 –1 1
3 –1 –1 1 1 1 –1 1
4 1 –1 –1 1 1 1 –1
4 1 –1 –1 1 1 1 –1
5 –1 1 –1 –1 1 1 1
5 –1 1 –1 –1 1 1 1
6 1 –1 1 –1 –1 1 1
6 1 –1 1 –1 –1 1 1
7 1 1 –1 1 –1 –1 1
7 1 1 –1 1 –1 –1 1
8 –1 –1 –1 –1 –1 –1 –1
Ave+ — — — — — — — —
Ave– — — — — — — — —
Effect — — — — — — — —
E1169 − 21
5.2.1 Main Effect Estimation:
5.2.1.1 For each factor column calculate the average of the test results corresponding to the + factor level and enter the result in
the Ave+ row in the column. Calculate the average of the test results corresponding to the – factor level and enter the result in the
Ave– row in the column.
5.2.1.2 Calculate the Main Effect = (Ave+ entry) minus (Ave– entry).
5.2.2 Half-Normal Plot—A half-normal plot is used to graphically identify potentially statistically significant active effects.
5.2.2.1 Construct a half-normal plot by plotting the absolute values of main effects on the X-axis, in order from smallest to largest,
against the half-normal plotting values given in Annex A2 on the Y-axis. Effects for all columns in the design, including columns
not used to assign levels to any real experiment factor, are plotted. The half-normal plotting values do not depend on the effect
values. They depend only on the half-normal distribution and the number of effects plotted.
5.2.2.2 If none of the factors have a measurable main effect, the plotted points will form a straight line. Factors having true effects
will lie to the right of the reference line formed by the smaller effects.
5.2.2.3 If an estimate of experimental error the standard deviation of test results (s ) ), which may be either repeatability or
etr
@ #
laboratory precision, is available, a reference line in the half-normal plot can be provided with slope1⁄s 51⁄ s 3 =~4 ⁄ N! slope 1
effect tr
/ . Effects s (see 5.2.3.2). Potentially significant effects are those that fall farthest to the right of the line.line may be considered
e
active effects.
5.2.2.4 If an estimate of error is not available, a reference line may be drawn by eye for the purpose of identifying potentially
significant effects. Select points with the smallest effects that appear to fall on a straight line from the origin. Draw a line starting
at the origin and passing close to the selected points.
NOTE 1—The slope of the line fit by eye does not provide a valid estimate of error when measurable effects appear to be present.
5.2.3 Statistical Significance Testing—If an estimate of precision is available or can be derived from the experiment (see Annex
A3), statistical tests of factor effects can be determined using the Student’s t-test. The t-test statistic for a factor is the main effect
divided by its standard error s , which is the same for all factors in a balanced and orthogonal design. If the t-value is greater than
e
the t-value corresponding to the 0.05 significance level, the factor is statistically significant at the 95 % 95 % confidence level.
5.2.3.1 If fewer factors are used with the design than the maximum number, then the “dummy effects” estimated for the unused
columns differ from zero only as a result of experimental error (or interactions of other factors). The root mean square of unused
effects is an estimate of the standard error of an effect having degrees of freedom equal to the number of unused effects averaged
(21).
1/2
5.2.3.2 In this case, the line described in 5.2.2.25.2.2.3 has slope 1 1 / (MSE)⁄ (MSE) , where MSE denotes the mean square
error of the dummy effects (see 5.1.1). For instance, if there are three dummy effects, e1,e1, e2,e2, and e3,e3, then MSE = (e1(e1
2 2
+ + e2e2 + + e3e3 ) ) / 3.⁄ 3.
5.2.4 Ruggedness Test Conclusions:
5.2.4.1 If no effects are identified as statistically significant and practically important, and if the experimenter is satisfied with the
way that the experiment was carried out and with its statistical power, then there is reason to think that the method is rugged with
regard to the factors tested.
5.2.4.2 If some effects are identified as statistically significant and practically important, then the method may have to be modified,
or specifications may need to be added for the range of acceptable values of the identified factors. In cases where the factor effects
may be statistically significant but not practically important the method can still be classified as “rugged.”
5.2.5 Supplemental Additions to the Basic Ruggedness Test Design:
5.2.5.1 The basic design allows only for the estimation of main effects. When there is uncertainty whether the factor effects change
E1169 − 21
with the levels of other factors in the experiment (43, 54), the main effects may be separated from the interaction effects by
conducting additional runs, as discussed in Section 6.
5.2.5.2 Each of the runs may be replicated to obtain an estimate of experimental variability in addition to that supplied by dummy
factors, and this may be conducted in three different ways, as discussed in Annex A3.
5.3 Example:
5.3.1 The example discussed here is part of a series of experiments that studied the effects of factors that influence determination
of pH in dilute acid solutions (43, 54). The factors and their levels are shown in Table 2. Factors C, D, and G are numerical, and
the rest are categorical. There are no dummy factors, so the design is said to be saturated (all columns assigned to factors).
5.3.2 The data and calculated main effects for the initial design are shown in Table 3. The results are recorded as 1000 pH.
5.3.3 In Table 3, the Ave+ value for factor A is the average of the four measurements at the “1” value for A (dilution of water):
3015, 2964, 2949 and 3055, the average of which is 2995.75. The Ave– value is the average of the four measurements at the “–1”
value for A (no dilution of water): 3006, 2999, 3049 and 2904, the average of which is 2989.5. The main effect is the difference
of these values 2995.75 – 2989.5 = 6.3. The other effect estimates are calculated analogously: B = 77.3, C = –0.8, D = 26.8, E
= 28.3, F = –1.3, G = 40.8.
5.3.4 Half-Normal Plot—The half-normal plotting values are shown in Table 4. As was statedsuggested in 5.2.25.2.2.4, a reference
line which passes through close to the three smallest values,values is added to the figure. From the half-normal plot in Fig. 1, we
see that factors B, G, E, and D appear to be significant. active.
5.3.5 Although the data discussed here give evidence of significantactive effects, that will not always happen. When no effects
appear significant,active, the method shows no evidence of lack of ruggedness. When there are significantactive effects, it may be
of value to do further experimentation to find significantactive two-factor interactions, as discussed in Section 6.
6. Separating Main Effects and Two-Factor Interactions with Added Foldover Runs
6.1 Interactions—If the effect of one factor depends on the level of another factor, then these two factors interact. As shown in
Section 5, a main effect for a factor is estimated by the difference between the mean measurement of the four high level
measurements and the mean of the four low level measurements. By contrast, the two-factor interaction between C and D is
estimated as follows. For the high level of factor C compute the difference (high level D mean – low level D mean), where each
mean is the average of two measurements. Calculate the corresponding difference at the low level. Half the difference of these two
differences is the interaction between factors C and D, in the sense that if the factor D effect does not vary by levels of factor C,
the two factor interaction calculated above should be close to 0. It turns out that the eight signs for Column C of Table 1, multiplied
by the corresponding eight signs in Column D, give a column of signs that specifies this same CD interaction if we take the
difference of the average of the measurements that correspond to “1” and the average of the measurements that correspond to “–1”.
In addition, it turns out that the negatives of these eight values is the same as the column A values. Thus, when we calculate the
factor A effect we are also calculating the negative of the CD interaction, written as –CD. There is no way to know whether the
main effect for A is really estimating that factor or the negative of the interaction between factors C and D.
6.1.1 Thus, the complication of the fractional factorial designs presented in Section 5 is that each main effect is confounded
(aliased) with a group of two-factor interactions, as shown in Table 5. Note that factor A is confounded with three two-way
interactions, one being –CD, which is discussed above. The other confounded interactions are –BF and –EG. Factors are said to
be “aliased” when their columns of signs are the negatives or positives of each other.
TABLE 2 Example: Factors That Influence Determination of pH in Dilute Acid Solutions
Factor No. Variable Units Level 1 (–) Level 2 (+)
A Dilution with water yes or no No yes
B Addition of potassium chloride yes or no No yes
C Equilibration time minutes 5 10
D Depth of electrode immersion cm 1 3
E Addition of sodium nitrate yes or no No yes
F Stirring yes or no No yes
G Temperature °C 2 4
E1169 − 21
TABLE 3 Results and Effects for Initial Design
PB Test
A B C D E F G
Order Result
1 1 1 1 –1 1 –1 –1 3015
1 1 1 1 –1 1 –1 –1 3015
2 –1 1 1 1 –1 1 –1 3006
2 –1 1 1 1 –1 1 –1 3006
3 –1 –1 1 1 1 –1 1 2999
3 –1 –1 1 1 1 –1 1 2999
4 1 –1 –1 1 1 1 –1 2964
4 1 –1 –1 1 1 1 –1 2964
5 –1 1 –1 –1 1 1 1 3049
5 –1 1 –1 –1 1 1 1 3049
6 1 –1 1 –1 –1 1 1 2949
6 1 –1 1 –1 –1 1 1 2949
7 1 1 –1 1 –1 –1 1 3055
7 1 1 –1 1 –1 –1 1 3055
8 –1 –1 –1 –1 –1 –1 –1 2904
Ave+ 2995.8 3031.3 2992.3 3006.0 3006.8 2992.0 3013.0
Ave– 2989.5 2954.0 2993.0 2979.3 2978.5 2993.3 2972.3
Main Effect 6.3 77.3 –0.8 26.8 28.3 –1.3 40.8
TABLE 4 Estimated Effects and Half-Normal Plotting Values
Half-Normal
Effect Order, e Effect Estimated Effect
Plotting Values
7 B 77.3 1.8
6 G 40.8 1.24
5 E 28.3 0.92
4 D 26.8 0.67
3 A 6.3 0.46
2 F 1.3 0.27
1 C 0.8 0.09
6.2 Design—To separate factor main effects from groups of two-factor interactions, the PB design is augmented with eight
additional runs called a foldover. A set of foldover runs is generated from the original design by changing all the 1’s to –1’s and
the –1’s to 1’s.Thus, 1’s. Thus, in the foldover, for each run, for each factor level of the initial design of Table 3, the opposite level
in each factor is used. As will be seen below, combining the two sets of runs will allow us to estimate the main effects without
confounding from the two-factor interactions. The set of foldover runs for the eight run PB design is shown in Table 6Table 6, ,
together with the test results and calculated main effects for these eight runs.
6.3 Analysis—To combine the results of original design and foldover in Table 7, the main effects are estimated by averaging the
main effect estimates from the two sets. The corresponding confounded interactions are estimated by taking half the difference of
the main effect estimates.
6.4 Half-Normal Plot—Using data from the initial runs and the foldover together, the effects are ordered by absolute value and
shown with the associated half-normal plot values in Table 8 and plotted in Fig. 2. The suffix –I, added to a factor label, indicates
the two factor interactions that are confounded with the factor. The nine smallest estimates appear to lie approximately on a straight
line, drawn in Fig. 2, thatfollowing 5.2.2.4represents the standard error for the estimates. The line was drawn to pass through the
nine smallest estimates approximately. . From the distribution of points in the plot, factors B, G, E, and D–I appear to be
statistically significant. The significance of active. Whether factor G–I is active is unclear.
6.5 In Table 5, it is shown that Interactions AF, CG, and DE are confounded with factor B. Thus, there is no way to know whether
the apparent significance of factor B is due to a confounded interaction. As a general rule, factors interact only when they have
large main effects in their own right. Hence, AF and CG are unlikely to be important, but a DE interaction could be contributing
to the estimated B effect. Similarly, AC, BE, and FG are confounded with D; a BE interaction could be contributing.
6.6 When factors are separated from confounded interactions, it appears that factor D is not significant,active, but the apparent
significance of D in the initial portion of the experiment was due to confounded interactions. The most likely cause of the large
D–I two-factor interaction is the BE interaction, since the main effects B and E are the largest, though only additional
experimentation can confirm this.
E1169 − 21
FIG. 1 Half-Normal Plot of pH Data
TABLE 5 Factorial Effect Aliases for Design in Table 1
[A] = A – BF – CD – EG
[B] = B – AF – CG – DE
[C] = C – AD – BG – EF
[D] = D – AC – BE – FG
[E] = E – AG – BD – CF
[F] = F – AB – CE – DG
[G] = G – AE – BC – DF
7. Keywords
7.1 foldover; fractional factorial design; half-normal plot; Plackett-Burman; ruggedness; ruggednessscreening design
E1169 − 21
TABLE 6 Results and Effects for Foldover Factor—Settings Are at the Opposite Level to the First Set (Table 3)
Test
PB Order A B C D E F G
Result
1 –1 –1 –1 1 –1 1 1 2931
1 –1 –1 –1 1 –1 1 1 2931
2 1 –1 –1 –1 1 –1 1 2978
2 1 –1 –1 –1 1 –1 1 2978
3 1 1 –1 –1 –1 1 –1 2967
3 1 1 –1 –1 –1 1 –1 2967
4 –1 1 1 –1 –1 –1 1 3030
4 –1 1 1 –1 –1 –1 1 3030
5 1 –1 1 1 –1 –1 –1 2874
5 1 –1 1 1 –1 –1 –1 2874
6 –1 1 –1 1 1 –1 –1 2979
6 –1 1 –1 1 1 –1 –1 2979
7 –1 –1 1 –1 1 1 –1 2911
7 –1 –1 1 –1 1 1 –1 2911
8 1 1 1 1 1 1 1 3040
8 1 1 1 1 1 1 1 3040
Ave+ 2964.8 3004.0 2963.8 2956.0 2977.0 2962.3 2994.8
Ave– 2962.8 2923.5 2963.8 2971.5 2950.5 2965.3 2932.8
Main Effect 2.0 80.5 0.0 –15.5 26.5 –3.0 62.0
TABLE 7 Calculation of Estimated Effects Using Data from
Table 3 and Table 6
Foldover
Factor Table 3 Average
(Table 6)
A 6.3 2.0 4.1
B 77.3 80.5 78.9
C –0.8 0.0 –0.4
D 26.8 –15.5 5.6
E 28.3 26.5 27.4
F –1.3 –3.0 –2.1
G 40.8 62.0 51.4
⁄2 difference
A–I = –BF – CD – EG 6.3 2.0 –2.1
B–I = –AF – CG – DE 77.3 80.5 1.6
C–I = –AD – BG – EF –0.8 0.0 0.38
D–I = –AC – BE – FG 26.8 –15.5 –21.1
E–I = –AG – BD – CF 28.3 26.5 –0.88
F–I = –AB – CE – DG –1.3 –3.0 –0.88
G–I = –AE – BC – DF 40.8 62.0 10.6
TABLE 8 Ordered Effects and Half-Normal Plotting Positions
Half-Normal
Factor Effect Abs (Effect)
Plotting Value
B 78.9 78.9 2.100
G 51.4 51.4 1.611
E 27.4 27.4 1.345
D–I –21.1 21.1 1.150
G–I 10.6 10.6 0.992
D 5.6 5.6 0.854
D 5.6 5.6 0.854
A 4.1 4.1 0.732
A 4.1 4.1 0.732
A–I –2.1 2.1 0.619
A–I –2.1 2.1 0.619
F –2.1 2.1 0.514
F –2.1 2.1 0.514
B–I 1.6 1.6 0.414
B–I 1.6 1.6 0.414
F–I –0.88 0.88 0.319
E–I –0.88 0.88 0.226
C–I 0.38 0.38 0.135
C –0.38 0.38 0.045
E1169 − 21
FIG. 2 Half-Normal Plot, Foldover pH Experiment
ANNEXES
(Mandatory Information)
A1. ADDITIONAL PLACKETT-BURMAN DESIGNS
A1.1 Plackett-Burman designs (1) are available for N values that are integer multiples of four. The following is a method for
constructing the designs for N = 4, 8, 12, 16, 20, and 24. The first row of each of these designs is given below for the associated
N value. Each row specifies which of the N – 1 factors will be set at the high level (1) or the low level (–1).
N = 4 1,1,–1
N = 8 1,1,1,–1,1,–1,–1
N = 12 1,1,–1,1,1,1,–1,–1,–1,1,–1
N = 16 1,1,1,1–1,1,–1,1,1,–1,–1,1,–1,–1,–1
N = 20 1,1,–1,–1,1,1,1,1,–1,1,–1,1,–1,–1,–1,–1,1,1,–1
N = 24 1,1,1,1,1–1,1,–1,1,1,–1,–1,1,1,–1,–1,1,–1,1,–1,–1,–1,–1
A1.2 For any selected N value, the corresponding set of N – 1 – 1 (1) and (–1) signs is written down as the first row of the design.
The second row of the design is obtained by copying the first row after shifting it one place to the right and putting the last sign
of Row 1 in the first position of Row 2. This type of cyclic shifting should be done a total of N – 2 times, after which a final row
of all minus signs is added. The result of this procedure for the N = 8 Plackett-Burman design is given in the first listed design
of this practice.
E1169 − 21
A2. PLOTTING POSITIONS FOR HALF-NORMAL PLOTS
A2.1 Table A2.1 gives the coordinates on the vertical axis for half-normal plots with k effects. The numerical estimates of all
effects, in order of smallest to largest, are the coordinates for the horizontal axis. Dummy effects, if present, are also plotted.
A2.2 k denotes the number of effects, and Φ(x) is the probability that the standard normal distribution gives a value less than x.
–1 –1
Φ (p) is the value x such that Φ(x) = p. If the k effects are arranged in order of increasing absolute value, the pairs (|effect |, Φ
e
–1
(0.5 + 0.5[e – 0.5] / k)) produce the appropriate half-normal plot for (e = 1,2,3,…k). See Ref (65). In Table A2.1, Φ (0.5 + 0.5[e
– 0.5] / k)) is denoted by H(e,k).
TABLE A2.1 Half-Normal Plotting Values (H(e,k) by Number of Effects (k) and Ordered Effects (e)
Number of 3 4 5 6 7 8 9 10 11
Effects, k
Ordered
effects,
smallest
to largest, e
1 0.210 0.157 0.126 0.105 0.090 0.078 0.070 0.063 0.057
2 0.674 0.489 0.385 0.319 0.272 0.237 0.210 0.189 0.172
3 1.383 0.887 0.674 0.549 0.464 0.402 0.355 0.319 0.289
4 1.534 1.036 0.812 0.674 0.579 0.508 0.454 0.410
5 1.645 1.150 0.921 0.776 0.674 0.598 0.538
6 1.732 1.242 1.010 0.862 0.755 0.674
7 1.803 1.318 1.085 0.935 0.825
8 1.863 1.383 1.150 0.998
9 1.915 1.440 1.207
10 1.960 1.489
11 2.000
Number of 12 13 14 15 16 17 18 19 20 21 22 23
Effects, k
Ordered
effects,
smallest
to largest, e
1 0.052 0.048 0.045 0.042 0.039 0.037 0.035 0.033 0.031 0.030 0.028 0.027
2 0.157 0.145 0.135 0.126 0.118 0.111 0.105 0.099 0.094 0.090 0.086 0.082
3 0.264 0.243 0.226 0.210 0.197 0.185 0.175 0.166 0.157 0.150 0.143 0.137
4 0.374 0.344 0.319 0.297 0.278 0.261 0.246 0.233 0.221 0.210 0.201 0.192
5 0.489 0.448 0.414 0.385 0.360 0.338 0.319 0.301 0.286 0.272 0.259 0.248
6 0.610 0.558 0.514 0.477 0.445 0.417 0.393 0.371 0.352 0.334 0.319 0.304
7 0.742 0.674 0.619 0.573 0.533 0.499 0.469 0.443 0.419 0.398 0.379 0.362
8 0.887 0.801 0.732 0.674 0.626 0.585 0.549 0.517 0.489 0.464 0.441 0.421
9 1.054 0.942 0.854 0.784 0.725 0.674 0.631 0.594 0.561 0.531 0.505 0.481
10 1.258 1.105 0.992 0.903 0.831 0.770 0.719 0.674 0.636 0.601 0.571 0.543
11 1.534 1.304 1.150 1.036 0.947 0.874 0.812 0.760 0.714 0.674 0.639 0.608
12 2.037 1.574 1.345 1.192 1.078 0.987 0.913 0.851 0.798 0.751 0.711 0.674
13 2.070 1.611 1.383 1.230 1.115 1.025 0.950 0.887 0.833 0.786 0.745
14 2.100 1.645 1.418 1.265 1.150 1.059 0.984 0.921 0.866 0.819
15 2.128 1.676 1.450 1.298 1.183 1.092 1.016 0.952 0.897
16 2.154 1.705 1.480 1.328 1.213 1.122 1.046 0.982
17 2.178 1.732 1.508 1.356 1.242 1.150 1.074
18 2.200 1.757 1.534 1.383 1.269 1.177
19 2.222 1.780 1.559 1.408 1.294
20 2.241 1.803 1.582 1.432
21 2.260 1.824 1.604
22 2.278 1.844
23 2.295
E1169 − 21
A3. COMPUTATIONS FOR DESIGNS WITH REPLICATED DATA
A3.1 Increasing the size of the experimentA replication of a test method is a complete repetition of all steps in the test method
on a new test unit to produce a new test result. Augmenting the basic design of Section 5 by replication of its N run conditions
improves the precision of factor effects and facilitates the allows for the estimation of experimental error and the evaluation of
statistical significance of the effects. The experimenter may simply duplicate runs at the same condition in successive runs, but this
may underestimate the experimental error. For statistical analysis, use the average of the repeats as data. Sensitivity of the
experiment can be increased using a fully randomized experiment with a replicate of each factor setting. Sensitivity is also
increased by the addition of a second block of runs that replicates the first (that is, runs with the same factor settings as the first
block). Resolution of the experiment can be increased using a second block of runs not replicating the first, such as a foldover.
The preferred practice is to use a foldover if there is uncertainty about the presence of two factor interactions.
A3.1.1 Each of the N run conditions in the basic design must be replicated by the same number, r, to achieve balance in a resulting
augmented design having rN runs. Usually r = 2, doubling the number of runs needed, and this level of replication is often sufficient
for statistical tests. The following calculations in this annex will be based on this level of replication. For r > 2, it is recommended
to consult a statistician.
A3.1.2 The experimental designs used for replication can be accomplished in two ways, resulting in either a fully replicated (FR)
design or a block replicated (BR) design.
A3.1.2.1 FR Design—In this design the 2N runs are conducted within the same experiment in random order (see 5.1.7). Statistical
tests on factor effects can be made available immediately following the conclusion of the experiment. The decision to whether to
use this augmented design or just the basic design with no replication must be made at the beginning of the ruggedness test.
NOTE A3.1—If the runs are not randomized, and the replicates for each run condition are conducted in succession (under repeatability conditions), the
experimental variation will usually be underestimated.
A3.1.2.2 BR Design—This design results when the basic design itself is repeated afterwards using the same run conditions. The
basic design and each repeat of it is termed a separate block. The second block of runs is executed independently of the first (basic
design) block, preferably in a new random order. It can be done at a different time period, or by using different operators. This
design can allow the basic design to be evaluated before a decision is made for additional blocks of runs.
A3.1.2.3 A ruggedness test example is given using r = 2 replicates for each of N = 8 run conditions, resulting in 16 total test results
used for the statistical analysis.
A3.2 In the case of simple duplication, duplicate determinations should be averaged and the averages plotted in a half-normal plot.
For the two other situations, the standard errors of the estimates can be computed from the following formula:
4s
tr
s 5 (A3.1)
Œ
effect
N 32
with degrees of freedom of (df),
where:
N = number of runs in the design, and
s = the estimated standard deviation of the test results.
tr
E1169 − 21
df = N for full randomization, (N – 1) for two blocks. The forms of df and s depend on the test design, and will be shown as
tr
each design is discussed.
A3.2 Example—The—An example of a seven-factor ruggedness experiment comes from a study done for Test Method F2082.
This test method determines a transformation temperature as the test result for nickel-titanium shape memory shape-memory
alloys. The factors of interest in the experimental design are quench method, bath temperature at deformation, equilibrium time,
bending strain, pin spacing, linear variable differential transducer (LVDT) probe weight, and heating rate. Table A3.1 provideslists
the descriptions and levels of the factors chosen in this example.
A3.2.1 There is no way of finding out which design, FR or BR, was actually used for this ruggedness test, so the calculations will
be shown for both designs.
A3.2.2 The calculations for the factor effects generally follow 5.2 for either design. The replication allows an estimate of the
experimental error to be calculated and then used for a statistical test for each factor. The calculations for the statistical analysis
used in this example are more straightforward for the FR design, and these will be given before the calculations are shown for the
BR design.
A3.4 After all tests are completed, the transformation temperature results are entered in Table A3.2 in the Rep1 and Rep2 Test
Result columns.
A3.5 The first calculations assume that the replicates are blocks. Calculations are made analogously to those in Table 3, but must
take the blocks into account. Factor main effects are calculated using the average values (Ave) of each design point for the two
blocks. At the bottom of each column are the averages of the replicate averages corresponding to the (1) and the averages of the
block averages corresponding to the (–1) signs in that column. For instance, in Table A3.2, for factor A, the (Ave+) value is the
average of measurements values corresponding to the (1 = water) signs in Column A: –27.29, –17.28, –31.70, and –15.44, which
yield an average of –22.93. The (Ave–) value is the average of the measurement values corresponding to the (–1 = air) signs in
Column A: –17.40, –27.76, –35.10, and –43.10, which average –30.84.
A3.6 The main effect row contains the difference [(Ave+) – (Ave–)] for that column. It may be interpreted as the result of changing
the factor shown in that column from low to high level. For factor A, since the Ave+ is 7.91 more than the Ave–, the effect is 7.91.
A3.3 Estimate the standard deviation of the test and the standard error of effects from the dispersion of differences (d , i = 1, 2,.
i
N) between replicates. The first pair of replicate readings is –26.95 (Rep1) and –27.63 (Rep2) and the difference (Rep2 – Rep1)
is –0.68. The remaining differences are: 0.74, 2.85, 1.15, –2.68, –2.55, 3.23, and –0.69. The standard deviation (s ) of the
d
2 th
differences is 2.23, where d is the i squared difference, and:
i
s 5 d ⁄ N 2 1 (A3.2)
~ !
d Π( i
@ #
~ !
i51,2,.N
A3.3 FR Design Calculations
where ∑ denotes the summation for i = 1,2,…N.
A3.3.1 Main Effects—Table A3.2 shows the design matrix for the seven factors A-G, having a similar format as Table 3 for the
basic design. The augmented design lists two columns for the test results, labeled Rep1 and Rep 2, a column for their Averages,
and a column for their Differences (Rep2 minus Rep1). The first pair of replicate readings is –26.95 (Rep1) and –27.63 (Rep2),
their Average is –27.29, and their Difference (Rep2 – Rep1) is –0.68. The factor Main Effects are calculated in the same way as
E1169 − 21
in Table 3 (see 5.2.1) except that the Average column is now used for these calculations. For factor A, the Ave+ is –22.93, the Ave–
is –30.84, and the Main Effect is 7.91. Using the averages instead of a column of 16 of individual test results eases the calculations
of Ave+ and Ave–.
A3.3.2 Standard Error of Effects—The experimental error variance, s , is estimated from the differences between replicates at each
tr
run condition.
Let d , i = 1, …, N, equal the difference between the replicates for the ith run condition. Calculate the variances of the N run
i
conditions as:
2 2
s 5 d ⁄2 (A3.1)
i i
These are listed for each run condition in the last column of Table A3.2. The estimated experimental error variance, s , with
tr
degrees of freedom = df = N, is:
N
2 2
s 5 s ⁄N (A3.2)
tr i
(
i51
In the FR example, the experimental error variance s = 2.19 and s = 1.48 with df = N = 8. The estimated standard error of
tr tr
the effect for either design FR or BR with r = 2 replicates: s is:
effect
2 2 2
4s 4s s
tr tr tr
s 5Π5Π5Π(A3.3)
effect
rN 238 4
1.48
=
In the FR example, s 5 2.19⁄45 50.74 with df = N = 8.
effect
A3.3.3 Statistical Tests on Effects—The t statistic is calculated for each effect as t = Main Effect / s and used to determine the
effect
p value, based on the number of degrees of freedom, df, for s . The t statistics and their p values are listed in Table A3.3. The
effect
factors D, A, B, and F are statistically significant at the 0.05 significance level. Factor C is borderline significant.
A3.8 The estimate of the standard deviation of the test results, s (see Section X1.1), is:
tr
s 2.23
d
s 5 5 5 1.58 (A3.3)
tr
1.414
=2
for the example data. For this example N = 8 in Eq A3.1, and:
4s
tr
s 5Π5 1.58/25 0.79, df 5 7 (A3.4)
effect
A3.9 Statistical significance of the factor effects and half-normal values for the half-normal plot are shown in Table A3.3:
A3.10 Dividing the effect by s provides a Student’s t-value, which has (N – 1) degrees of freedom (Section A3.1), seven
effect
degrees of freedom for this experiment for both s and Student’s t. For example, for Effect A, the t-value is 7.91 / 0.79 = 10.04.
effect
Based on the assumption that there is no effect, the probability of a t score as large as 10.04 is approximately 0 (p-value < 0.001).
From the t-test results, the factors D, A, B, and F are statistically significant.
A3.11 Alternatively, the same estimated effects are shown in Table A3.4 as in Table A3.3, but are treated as coming from a full
randomization. In this case, the estimated standard error is calculated differently from that in two blocks. For fully randomized
data, the test result variance is the average of the eight sample variances calculated from the two repeats of each of the test
conditions. Each sample variance is half the squared difference of the repeats. For example, the sample variance for trial 1 test
2 2
conditions of Table A3.2 is [(–27.63) – (–26.95)] / 2 = (–0.68) / 2 = 0.23. The seven other variances are computed analogously
and are 0.27, 4.06, 0.66, 3.59, 3.25, 5.22, 0.24. The average is 2.19 with square root s = 1.48:
st
s 5 squa
...