ISO/TS 27878:2023
(Main)Reproducibility of the level of detection (LOD) of binary methods in collaborative and in-house validation studies
Reproducibility of the level of detection (LOD) of binary methods in collaborative and in-house validation studies
This document provides statistical techniques for the determination of the reproducibility of the level of detection for a) binary (qualitative) test methods for continuous measurands, e.g. the content of a chemical substance, and b) binary (qualitative) test methods for discrete measurands, e.g. the number of RNA copies in a sample. The reproducibility precision is determined according to ISO 5725 (all parts). Precision estimates are subject to random variability. Accordingly, it is important to determine the uncertainty associated with each estimate, and to understand the relationship between this uncertainty, the number of participants and the experimental design. This document thus provides not only a description of statistical tools for the calculation of the LOD reproducibility precision, but also for the standard error of the estimates.
Reproductibilité du niveau de détection (LOD) des méthodes binaires pour des études de validation collaboratives et internes
Le présent document fournit des techniques statistiques pour la détermination de la reproductibilité du niveau de détection pour: a) les méthodes d’essai binaires (qualitatives) pour les mesurandes continus, par exemple dans le contenu d’une substance chimique; b) les méthodes d’essai binaires (qualitatives) pour les mesurandes discrets, par exemple dans le nombre de copies d’ARN dans un échantillon. La fidélité de la reproductibilité est déterminée conformément à l’ISO 5725 (toutes les parties). Les estimations de la fidélité sont sujettes à une variabilité aléatoire. Par conséquent, il est important de déterminer l’incertitude associée à chaque estimation et de comprendre la relation entre cette incertitude, le nombre de participants et le plan d’expérience. À cet effet, le présent document décrit les outils statistiques non seulement pour le calcul de la fidélité de la reproductibilité de LOD, mais aussi pour l’erreur-type des estimations.
General Information
Relations
Standards Content (Sample)
TECHNICAL ISO/TS
SPECIFICATION 27878
First edition
2023-01
Reproducibility of the level of
detection (LOD) of binary methods in
collaborative and in-house validation
studies
Reproductibilité de la limite de détection (LD) des méthodes binaires
pour des études de validation internes et collaboratives
Reference number
ISO/TS 27878:2023(E)
© ISO 2023
---------------------- Page: 1 ----------------------
ISO/TS 27878:2023(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO 2023
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii
© ISO 2023 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/TS 27878:2023(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Symbols . 2
5 General principles . 3
5.1 General considerations. 3
5.2 Considerations for the conventional approach . 3
5.3 Considerations for the factorial approach . 3
6 Conventional approach . 4
6.1 Experimental design . 4
6.2 Statistical model for methods for continuous measurands. 4
6.3 Statistical model for methods for discrete measurands . 7
6.4 Reliability of precision estimates . 10
7 Factorial approach .10
8 In-house validation .13
9 Software .13
Bibliography .15
iii
© ISO 2023 – All rights reserved
---------------------- Page: 3 ----------------------
ISO/TS 27878:2023(E)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see
www.iso.org/iso/foreword.html.
This document was prepared by Technical Committee ISO/TC 69, Applications of statistical methods,
Subcommittee SC 6, Measurement methods and results.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www.iso.org/members.html.
iv
© ISO 2023 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/TS 27878:2023(E)
Introduction
An appropriate approach for the validation of binary methods will often differ considerably from that
of quantitative methods. Nevertheless, core concepts from the validation of quantitative methods can
be successfully carried over to binary methods. In particular, the precision of a method – a performance
characteristic usually associated with quantitative methods – can be determined for the level of
detection (LOD) of binary methods.
In analytical chemistry, one of the fundamental indicators of method performance is the reproducibility
[1]
of quantitative test results as described in ISO 5725 (all parts) . This aspect of method performance is
not usually taken into consideration in the validation of binary methods. However, in the last few years,
novel validation approaches have been proposed in which the reproducibility of a binary method can be
determined and meaningfully interpreted.
Why is it important to determine a method’s reproducibility? In order to answer this question, consider
an example from the field of microbiology. Take the case that, in the validation study, a method’s LOD
is determined as 3 CFU/ml (CFU = colony forming unit), but that the LOD is sometimes much higher
depending on the laboratory or on the test conditions. Failing to detect the occasional unreliability of
the method could lead to mistakes in routine laboratory determinations. On the other hand, if an LOD
of 300 CFU/ml is obtained in the validation study, the method will not be validated even though this
excessive LOD is not representative of its average performance. Accordingly, both the average LOD
value and the reproducibility parameter – describing the variability of the LOD across laboratories or
test conditions – capture important information about the performance of the method and should be
determined in the course of the validation process.
In order to accomplish this, a suitable approach should be identified for the conversion of the binary
results into quantitative ones. In this standard, two parametric models for the calculation of the LOD
will be used: one model for methods for discrete measurands, e.g. microbiological and Polymerase
Chain Reaction (PCR) methods, and one model for methods for continuous measurands, e.g. chemical
methods.
Two different study designs will be applied. In the conventional approach, test conditions vary randomly
from one laboratory to the other, whereas in the factorial approach, at least to some extent, test
conditions are controlled. The factorial approach makes it possible to assess different sources of errors
such as the variability arising in connection with different analysts, different instruments, different
lots of reagents, different elapsed assay times, different assay temperatures etc. Such an approach also
allows a reduction in workload and fewer participating laboratories.
v
© ISO 2023 – All rights reserved
---------------------- Page: 5 ----------------------
TECHNICAL SPECIFICATION ISO/TS 27878:2023(E)
Reproducibility of the level of detection (LOD) of binary
methods in collaborative and in-house validation studies
1 Scope
This document provides statistical techniques for the determination of the reproducibility of the level
of detection for
a) binary (qualitative) test methods for continuous measurands, e.g. the content of a chemical
substance, and
b) binary (qualitative) test methods for discrete measurands, e.g. the number of RNA copies in a
sample.
The reproducibility precision is determined according to ISO 5725 (all parts).
Precision estimates are subject to random variability. Accordingly, it is important to determine
the uncertainty associated with each estimate, and to understand the relationship between this
uncertainty, the number of participants and the experimental design. This document thus provides not
only a description of statistical tools for the calculation of the LOD reproducibility precision, but also
for the standard error of the estimates.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO 3534-1, Statistics — Vocabulary and symbols — Part 1: General statistical terms and terms used in
probability
ISO 5725-1, Accuracy (trueness and precision) of measurement methods and results — Part 1: General
principles and definitions
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO 3534-1 and ISO 5725-1 and
the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
factor
binary or quantitative parameter within the method that can be varied at two or more levels within the
limits of the specified method
EXAMPLE Technician.
1
© ISO 2023 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/TS 27878:2023(E)
3.2
factor level
value of the factors (3.1) within the experimental design
EXAMPLE Technician 1, Technician 2, etc.
3.3
level of detection
LOD
concentration from which on the POD (3.4) is not below a specified limit, e.g. 0,5 or 0,95 (LOD or
50%
LOD ).
95%
Note 1 to entry: This definition is mathematically equivalent to the definitions for “level of detection” in
[2] [3] [4]
ISO 16140-1 , ISO 16140-2 and ISO 16140-4 . It differs from the definition used for chemical and physical
methods for which a “limit of detection” is defined as the lowest quantity of an analyte that can be distinguished
from the absence of that analyte with a stated confidence level.
Note 2 to entry: In this document, the term concentration (or concentration level) is used as a generic term to
mean not only the actual concentration in the case of a measurand that can be quantified on a continuous scale,
but also the number of colony forming units or DNA copies per aliquot in the case of measurands which are
quantified on a discrete scale.
3.4
probability of detection
POD
probability of a positive analytical outcome of a qualitative test method at a given concentration for a
specific sample type
Note 1 to entry: This definition is based on the two slightly different definitions for “probability of detection” in
[6]
ISO/TS 16393 and ISO 16140-1, ISO 16140-2 and ISO 16140-4.
Note 2 to entry: The POD is a measure of the probability of a positive analytical result and thus a theoretical
value which can be approximated by a mathematical model.
3.5
rate of detection
ROD
proportion of positive analytical outcomes in a test series, when a qualitative method is performed
several times with a specific sample
Note 1 to entry: The ROD is not a theoretical value based on a mathematical model [like the POD (3.4)] but the
result of a series of repeated tests performed on a given sample.
4 Symbols
p number of participating laboratories
2
between-laboratory variance
σ
L
POD = P probability of detection
x concentration level (see Note 1 to entry 3.3) at which the POD is calculated
ROD rate of detection
LOD = L 50 % of the level of detection
50% 50
LOD = L 95 % of the level of detection
95% 95
L, H, B, C global model parameters for the four-parameter sigmoid curve
2
© ISO 2023 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/TS 27878:2023(E)
a laboratory-specific correction of laboratory i for the global inflection point C
i
2
2
normal distribution with mean μ and variance σ
N μσ,
()
5 General principles
5.1 General considerations
In order to ensure that tests are conducted in the same manner in all participating laboratories, the test
method should be standardized. All tests forming part of an experiment within an individual laboratory
or of an interlaboratory experiment shall be carried out according to the corresponding standardized
protocol.
The statistical methods described in this document are applicable for binary test methods which yield
a yes/no result (e.g. the substance of interest is present or absent). For such test methods, one of the
main criteria of the method’s fitness for purpose is the level of detection (e.g. LOD or LOD ), i.e.
50% 95%
the (concentration) level required to ensure a POD of 50 % or 95 %. The aim is thus to determine LOD
values for the individual laboratories as well as an overall LOD across laboratories. The precision of the
method can then be evaluated in terms of the variability to which the laboratory-specific LOD values
are subjected.
The laboratory-specific LOD values and the mean LOD across laboratories can be computed based on a
mathematical model for the relationship between level, x, and probability of detection POD xP= ()x
()
ii
for laboratory i: The LOD of laboratory i is then the lowest level, x, for which POD xP=≥()x 09, 5
()
95% ii
.
5.2 Considerations for the conventional approach
The conventional approach is based on the assumption that, according to the design used in ISO 5725-2,
all tests are performed under repeatability conditions in each of the laboratories involved. In particular,
all tests in the laboratory are performed by the same technician, with the same equipment, under the
same conditions and directly one after the other. Test results are considered to have been obtained
from different laboratories under reproducibility conditions, i.e. many factors contribute to observed
variability, e.g. differences in equipment, environmental conditions, reagent batches or technician.
NOTE Validation protocols according to the conventional approach based on LOD and POD can be found
[7]
in ISO 16140-2, ISO 16140-4 and ISO/TS 16393 and AOAC Guidelines . Examples and further protocols are
discussed e.g. in References [8][9][10][11][12] and [13].
5.3 Considerations for the factorial approach
Compared to the conventional approach, in which tests are made under repeatability conditions in each
of the laboratories, the factorial approach systematically varies one or more factors. For instance, half
the tests are conducted with reagents from batch A, and the other half with reagents from batch B.
Thus, the factorial approach makes it possible to ensure the full spectrum of test conditions is covered
in the validation study and assess contributions to variability from separate sources of error. This
approach translates to more efficient and reliable estimation of the total variability.
NOTE Validation protocols based on LOD for microbiological methods according to the factorial approach
[4] [5]
are given in ISO 16140-4 and ISO 16140-5 .
3
© ISO 2023 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/TS 27878:2023(E)
6 Conventional approach
6.1 Experimental design
Results from at least 8 participants, 4 concentration levels, and 8 replicates per level and laboratory
are required to obtain a statistically reliable POD curve. However, with such a design, the reliability
of the results may not be sufficient and will need to be checked. For more reliable estimation of the
LOD and the corresponding variability, it is recommended that results from at least 8 participants,
5 concentration levels, and 12 replicates per level and laboratory are available. If the number of
participants is increased, the number of replicates can be reduced.
The lowest concentration level should be selected so that no further reduction in POD is expected,
even if the concentration level is further reduced. The highest concentration level should be selected in
such a way that no further increase in POD is to be expected even if the concentration level is further
increased. The expected proportions of positive test results across laboratories should be between
20 % and 80 % for at least two concentration levels.
The proportion of positive test results expected at the beginning of the collaborative trial usually differs
from the final POD. This may mean that the proportion of positive test results actually determined in
the collaborative trial does not meet the above requirements. In this case, the results of the evaluation
and, in particular, the calculated reproducibility of the LOD can only be regarded as an estimate.
NOTE These recommendations for the experimental design are based on simulation studies in which the
standard error of the estimate of the laboratory standard deviation was evaluated.
6.2 Statistical model for methods for continuous measurands
The calculation of the LOD is based on a generalized linear mixed-effects model (GLMM) together with
a four-parameter sigmoid curve given by Formula (1):
LH−
POD=P = +H (1)
ii
B
x
1+
aC
i
where
i denotes the laboratory (i = 1, 2,., p);
POD = P denote the probability of detection for laboratory i;
i i
x denotes a given concentration level;
L, H, B, C are global model parameters (i.e. they are valid across all laboratories);
a denotes the laboratory-specific correction of laboratory i;
i
C denotes the global inflection point C.
It is assumed that the parameters, L (lowest probability of detection), H (highest probability of
detection), and B (slope) are the same for all laboratories. The product a C describes the location of the
i
inflection point of the curve for laboratory i; for L = 0 %, H = 100 %, it corresponds to the concentration
at which a POD of 50 % is reached. The value of this product is thus a direct measure of the performance
of the specific laboratory. The parameter, C, corresponds to the performance of an average laboratory.
4
© ISO 2023 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/TS 27878:2023(E)
The a values are modelled as realizations of a random variable: It is assumed that the ln a values follow
i i
a normal distribution with
2
lna ∼N 0,σ
()
i L
2
The parameters L, H, B, C and σ can be provided by maximum likelihood estimation, e.g. in
L
2
mathematical-statistical software package. The variance σ characterizes the variability of sensitivity
L
between laboratories.
NOTE 1 Although there is no guarantee that the distribution of ln a values actually follows a normal
i
distribution, the log transformation usually leads to a better approximation of the normal distribution. If the
method displays poor precision, then the prediction range of the LOD values without log transformation could
include infeasible negative values.
NOTE 2 It is assumed that the parameters L, H, C and B are the same for all laboratories, i.e. that the shape
of the curve is sigmoidal and the same across laboratories. It should be checked whether this assumption is
justified, e.g. through a graphic check of laboratory-specific POD curves.
The interpretation of the parameters will be explained with an example, see Reference [13]. A
collaborative study of a method for the binary analysis of gluten in corn products was conducted to
demonstrate that the binary test method can detect gluten contaminations below the threshold of
20 mg/kg gluten. A total of four corn sample lots with different gluten concentrations was submitted
to 18 laboratories to evaluate the sensitivity and reproducibility of the test method. Each of the 18
laboratories conducted 10 tests for each of four concentration levels. Table 1 provides the corresponding
numbers of positive results per laboratory and concentration level.
Table 1 — Number of positive test results per concentration level and laboratory (10 replicates)
Concentration level
Laboratory
No.
0,88 mg/kg 2,42 mg/kg 5,48 mg/kg 9,38 mg/kg
01 0 10 10 10
02 0 10 10 10
03 0 10 10 10
04 0 10 10 10
05 0 10 10 10
06 0 10 10 10
07 0 10 10 10
08 0 9 10 10
09 0 10 10 10
10 0 9 8 10
11 0 10 10 10
12 0 10 10 10
13 0 10 10 10
14 0 10 10 10
15 0 9 10 10
16 0 10 10 10
17 0 10 10 10
18 2 10 10 10
Figure 1 shows the POD curve of a laboratory with average performance (solid line) along with 95 %
prediction range of laboratory-specific POD (dark grey zone) and 95 % prediction range of laboratory-
specific RODs (light grey step-functions). The numbers adjacent to the diamonds indicate the
laboratory numbers having obtained the corresponding ROD.
5
© ISO 2023 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/TS 27878:2023(E)
For instance, at the concentration level 0,88 mg/kg, one laboratory has an ROD of 0,2, and 17
laboratories have an ROD of 0. Comparison with Table 1 shows that the laboratory with the ROD of
0,2 is laboratory 18. The light grey step-functions show the 95 % prediction range for the ROD values,
obtained from simulation runs performed on the basis of the parameter estimates (Monte Carlo
simulation). Figure 1 can be read as follows: a POD of 80 % is reached by a laboratory with an average
performance at a concentration of about 1,7 mg/kg (solid line), whereas a top-performing laboratory
will reach this POD at 1,3 mg/kg (upper dark grey zone) and a low-performing laboratory will need a
concentration of about 2,2 mg/kg (lower dark grey zone).
NOTE 3 None of the selected concentration levels is within the 20 % to 80 % interval; therefore, the calculated
reproducibility data can only be considered as an inaccurate estimate.
Key
X concentration, in mg/kg Y POD and ROD
Figure 1 — Mean POD curve, laboratory-specific RODs and prediction ranges
A special case of the model in 6.2 with L = 0, H = 1 and constant a value is equivalent to the logit
i
model for x > 0. In other words, the logit model is already included in the model in 6.2. In practical
terms, this statement also holds for the probit model, since it is very similar to the logit model, see e.g.
Reference [14].
If continuous test results are available, the validation study should be based on these rather than on
the corresponding binary results. In other words, insofar as binary results are obtained by comparing
continuous test results to a threshold, the laboratories should submit the original continuous results,
and the comparison with the threshold should be conducted as part of the validation study.
In many cases, the original continuous results will not be available, of course. In particular, in many
cases, the assay yields a binary result, even though it is based on a continuous response.
2
Finally, it should be noted that the estimate of the between-laboratory variance σ obtained from the
L
binary results on the basis of the model described above is closely related to the between-laboratory
standard deviation σ from ISO 5725-2. Indeed, if p laboratories each submitted two replicate LOD
L
6
© ISO 2023 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/TS 27878:2023(E)
values in a collaborative study, it would be possible to consider the σ estimate computed according to
L
ISO 5725-2 to be equivalent to the σ estimate as computed here.
L
2
NOTE 4 Given an estimate for a variance (such as the between-laboratory variance estimate σ mentioned
L
above), the corresponding standard deviation is obtained by taking the square root.
6.3 Statistical model for methods for discrete measurands
In the case of measurands quantified on a discrete scale (e.g. microbiological culture methods or
PCR methods), the four-parameter model discussed in 6.2 is no longer appropriate. The reason is the
difference in distributional assumptions regarding the concentration, x. In 6.2, x denotes a nominal
concentration level per se, and differences between the actual concentration of a test portion and
the nominal concentration level can be assumed to be negligible. In the case of discrete measurands,
x denotes e.g. the number of colony-forming units or DNA copies per test portion. For the sake of
terminological convenience, these discrete quantities are referred to as concentration levels (see Note 1
to entry to definition 3.3) but, in the case of the discrete measurands considered here, differences
between the actual concentration in a test portion and the nominal concentration can no longer be
assumed to be negligible; rather, for a given nominal concentration level, the actual concentration levels
of test portions are assumed to be subject to random variability and to follow a Poisson distribution.
This assumption will be referred to in the following as the “Poisson assumption”. For this reason, the
cloglog (complementary-log-log) model is appropriate for the calculation of the LOD and its variability
in the case of discrete measurands; accordingly, the following generalized mixed linear model (GLMM)
is applied:
ln{}−−ln[]1POD ()xa=+ln bxln
i i
where
i denotes the laboratory (i = 1,2,.,p);
x denotes a given concentration level;
b is a global positive parameter that models the dependence of the sensitivity on the concentration
level;
a denotes the sensitivity corresponding to laboratory i.
i
NOTE 1 The Poisson assumption requires that POD ()xP==0 for x= 0 . This means that the above model
ii
should only be used if the number of false-positive results is negligible. Another consequence of the Poisson
assumption is that POD ()xP= ()x will approach 1 with increasing x; in other words, the model is also
ii
susceptible to false negatives. This assumption constitutes an important difference to the four-parameter model
discussed in 6.2, which admits both false positives and false negatives.
NOTE 2 The complementary log-log model is a standard model for microbiological methods and qualitative
PCR. The model establishes a relationship between the probability of a positive result and the concentration,
when a test portion is taken from a homogeneous sample. It is assumed that the probability of detecting an
individual cell or DNA (RNA) copy does not depend on the concentration level. The probability of a positive result
is then simply derived from the Poisson distribution: a qualitative result is positive if at least one cell or DNA
(RNA) copy is detected.
It is assumed that the ln a values follow a normal distribution with
i
2
lnaμ∼N μσ,, = lna.
()
i L
2
The three parameters a, σ and b can be determined by maximum likelihood estimation in standard
L
statistical software such as R. The parameter a represents the average sensitivity parameter (at x
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.