# ISO/TS 27878:2023

(Main)## Reproducibility of the level of detection (LOD) of binary methods in collaborative and in-house validation studies

## Reproducibility of the level of detection (LOD) of binary methods in collaborative and in-house validation studies

This document provides statistical techniques for the determination of the reproducibility of the level of detection for a) binary (qualitative) test methods for continuous measurands, e.g. the content of a chemical substance, and b) binary (qualitative) test methods for discrete measurands, e.g. the number of RNA copies in a sample. The reproducibility precision is determined according to ISO 5725 (all parts). Precision estimates are subject to random variability. Accordingly, it is important to determine the uncertainty associated with each estimate, and to understand the relationship between this uncertainty, the number of participants and the experimental design. This document thus provides not only a description of statistical tools for the calculation of the LOD reproducibility precision, but also for the standard error of the estimates.

## Reproductibilité du niveau de détection (LOD) des méthodes binaires pour des études de validation collaboratives et internes

Le présent document fournit des techniques statistiques pour la détermination de la reproductibilité du niveau de détection pour: a) les méthodes d’essai binaires (qualitatives) pour les mesurandes continus, par exemple dans le contenu d’une substance chimique; b) les méthodes d’essai binaires (qualitatives) pour les mesurandes discrets, par exemple dans le nombre de copies d’ARN dans un échantillon. La fidélité de la reproductibilité est déterminée conformément à l’ISO 5725 (toutes les parties). Les estimations de la fidélité sont sujettes à une variabilité aléatoire. Par conséquent, il est important de déterminer l’incertitude associée à chaque estimation et de comprendre la relation entre cette incertitude, le nombre de participants et le plan d’expérience. À cet effet, le présent document décrit les outils statistiques non seulement pour le calcul de la fidélité de la reproductibilité de LOD, mais aussi pour l’erreur-type des estimations.

### General Information

### Relations

### Standards Content (Sample)

TECHNICAL ISO/TS

SPECIFICATION 27878

First edition

2023-01

Reproducibility of the level of

detection (LOD) of binary methods in

collaborative and in-house validation

studies

Reproductibilité de la limite de détection (LD) des méthodes binaires

pour des études de validation internes et collaboratives

Reference number

ISO/TS 27878:2023(E)

© ISO 2023

---------------------- Page: 1 ----------------------

ISO/TS 27878:2023(E)

COPYRIGHT PROTECTED DOCUMENT

© ISO 2023

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on

the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below

or ISO’s member body in the country of the requester.

ISO copyright office

CP 401 • Ch. de Blandonnet 8

CH-1214 Vernier, Geneva

Phone: +41 22 749 01 11

Email: copyright@iso.org

Website: www.iso.org

Published in Switzerland

ii

© ISO 2023 – All rights reserved

---------------------- Page: 2 ----------------------

ISO/TS 27878:2023(E)

Contents Page

Foreword .iv

Introduction .v

1 Scope . 1

2 Normative references . 1

3 Terms and definitions . 1

4 Symbols . 2

5 General principles . 3

5.1 General considerations. 3

5.2 Considerations for the conventional approach . 3

5.3 Considerations for the factorial approach . 3

6 Conventional approach . 4

6.1 Experimental design . 4

6.2 Statistical model for methods for continuous measurands. 4

6.3 Statistical model for methods for discrete measurands . 7

6.4 Reliability of precision estimates . 10

7 Factorial approach .10

8 In-house validation .13

9 Software .13

Bibliography .15

iii

© ISO 2023 – All rights reserved

---------------------- Page: 3 ----------------------

ISO/TS 27878:2023(E)

Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards

bodies (ISO member bodies). The work of preparing International Standards is normally carried out

through ISO technical committees. Each member body interested in a subject for which a technical

committee has been established has the right to be represented on that committee. International

organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.

ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of

electrotechnical standardization.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of

any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www.iso.org/patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to

the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see

www.iso.org/iso/foreword.html.

This document was prepared by Technical Committee ISO/TC 69, Applications of statistical methods,

Subcommittee SC 6, Measurement methods and results.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www.iso.org/members.html.

iv

© ISO 2023 – All rights reserved

---------------------- Page: 4 ----------------------

ISO/TS 27878:2023(E)

Introduction

An appropriate approach for the validation of binary methods will often differ considerably from that

of quantitative methods. Nevertheless, core concepts from the validation of quantitative methods can

be successfully carried over to binary methods. In particular, the precision of a method – a performance

characteristic usually associated with quantitative methods – can be determined for the level of

detection (LOD) of binary methods.

In analytical chemistry, one of the fundamental indicators of method performance is the reproducibility

[1]

of quantitative test results as described in ISO 5725 (all parts) . This aspect of method performance is

not usually taken into consideration in the validation of binary methods. However, in the last few years,

novel validation approaches have been proposed in which the reproducibility of a binary method can be

determined and meaningfully interpreted.

Why is it important to determine a method’s reproducibility? In order to answer this question, consider

an example from the field of microbiology. Take the case that, in the validation study, a method’s LOD

is determined as 3 CFU/ml (CFU = colony forming unit), but that the LOD is sometimes much higher

depending on the laboratory or on the test conditions. Failing to detect the occasional unreliability of

the method could lead to mistakes in routine laboratory determinations. On the other hand, if an LOD

of 300 CFU/ml is obtained in the validation study, the method will not be validated even though this

excessive LOD is not representative of its average performance. Accordingly, both the average LOD

value and the reproducibility parameter – describing the variability of the LOD across laboratories or

test conditions – capture important information about the performance of the method and should be

determined in the course of the validation process.

In order to accomplish this, a suitable approach should be identified for the conversion of the binary

results into quantitative ones. In this standard, two parametric models for the calculation of the LOD

will be used: one model for methods for discrete measurands, e.g. microbiological and Polymerase

Chain Reaction (PCR) methods, and one model for methods for continuous measurands, e.g. chemical

methods.

Two different study designs will be applied. In the conventional approach, test conditions vary randomly

from one laboratory to the other, whereas in the factorial approach, at least to some extent, test

conditions are controlled. The factorial approach makes it possible to assess different sources of errors

such as the variability arising in connection with different analysts, different instruments, different

lots of reagents, different elapsed assay times, different assay temperatures etc. Such an approach also

allows a reduction in workload and fewer participating laboratories.

v

© ISO 2023 – All rights reserved

---------------------- Page: 5 ----------------------

TECHNICAL SPECIFICATION ISO/TS 27878:2023(E)

Reproducibility of the level of detection (LOD) of binary

methods in collaborative and in-house validation studies

1 Scope

This document provides statistical techniques for the determination of the reproducibility of the level

of detection for

a) binary (qualitative) test methods for continuous measurands, e.g. the content of a chemical

substance, and

b) binary (qualitative) test methods for discrete measurands, e.g. the number of RNA copies in a

sample.

The reproducibility precision is determined according to ISO 5725 (all parts).

Precision estimates are subject to random variability. Accordingly, it is important to determine

the uncertainty associated with each estimate, and to understand the relationship between this

uncertainty, the number of participants and the experimental design. This document thus provides not

only a description of statistical tools for the calculation of the LOD reproducibility precision, but also

for the standard error of the estimates.

2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO 3534-1, Statistics — Vocabulary and symbols — Part 1: General statistical terms and terms used in

probability

ISO 5725-1, Accuracy (trueness and precision) of measurement methods and results — Part 1: General

principles and definitions

3 Terms and definitions

For the purposes of this document, the terms and definitions given in ISO 3534-1 and ISO 5725-1 and

the following apply.

ISO and IEC maintain terminology databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp

— IEC Electropedia: available at https:// www .electropedia .org/

3.1

factor

binary or quantitative parameter within the method that can be varied at two or more levels within the

limits of the specified method

EXAMPLE Technician.

1

© ISO 2023 – All rights reserved

---------------------- Page: 6 ----------------------

ISO/TS 27878:2023(E)

3.2

factor level

value of the factors (3.1) within the experimental design

EXAMPLE Technician 1, Technician 2, etc.

3.3

level of detection

LOD

concentration from which on the POD (3.4) is not below a specified limit, e.g. 0,5 or 0,95 (LOD or

50%

LOD ).

95%

Note 1 to entry: This definition is mathematically equivalent to the definitions for “level of detection” in

[2] [3] [4]

ISO 16140-1 , ISO 16140-2 and ISO 16140-4 . It differs from the definition used for chemical and physical

methods for which a “limit of detection” is defined as the lowest quantity of an analyte that can be distinguished

from the absence of that analyte with a stated confidence level.

Note 2 to entry: In this document, the term concentration (or concentration level) is used as a generic term to

mean not only the actual concentration in the case of a measurand that can be quantified on a continuous scale,

but also the number of colony forming units or DNA copies per aliquot in the case of measurands which are

quantified on a discrete scale.

3.4

probability of detection

POD

probability of a positive analytical outcome of a qualitative test method at a given concentration for a

specific sample type

Note 1 to entry: This definition is based on the two slightly different definitions for “probability of detection” in

[6]

ISO/TS 16393 and ISO 16140-1, ISO 16140-2 and ISO 16140-4.

Note 2 to entry: The POD is a measure of the probability of a positive analytical result and thus a theoretical

value which can be approximated by a mathematical model.

3.5

rate of detection

ROD

proportion of positive analytical outcomes in a test series, when a qualitative method is performed

several times with a specific sample

Note 1 to entry: The ROD is not a theoretical value based on a mathematical model [like the POD (3.4)] but the

result of a series of repeated tests performed on a given sample.

4 Symbols

p number of participating laboratories

2

between-laboratory variance

σ

L

POD = P probability of detection

x concentration level (see Note 1 to entry 3.3) at which the POD is calculated

ROD rate of detection

LOD = L 50 % of the level of detection

50% 50

LOD = L 95 % of the level of detection

95% 95

L, H, B, C global model parameters for the four-parameter sigmoid curve

2

© ISO 2023 – All rights reserved

---------------------- Page: 7 ----------------------

ISO/TS 27878:2023(E)

a laboratory-specific correction of laboratory i for the global inflection point C

i

2

2

normal distribution with mean μ and variance σ

N μσ,

()

5 General principles

5.1 General considerations

In order to ensure that tests are conducted in the same manner in all participating laboratories, the test

method should be standardized. All tests forming part of an experiment within an individual laboratory

or of an interlaboratory experiment shall be carried out according to the corresponding standardized

protocol.

The statistical methods described in this document are applicable for binary test methods which yield

a yes/no result (e.g. the substance of interest is present or absent). For such test methods, one of the

main criteria of the method’s fitness for purpose is the level of detection (e.g. LOD or LOD ), i.e.

50% 95%

the (concentration) level required to ensure a POD of 50 % or 95 %. The aim is thus to determine LOD

values for the individual laboratories as well as an overall LOD across laboratories. The precision of the

method can then be evaluated in terms of the variability to which the laboratory-specific LOD values

are subjected.

The laboratory-specific LOD values and the mean LOD across laboratories can be computed based on a

mathematical model for the relationship between level, x, and probability of detection POD xP= ()x

()

ii

for laboratory i: The LOD of laboratory i is then the lowest level, x, for which POD xP=≥()x 09, 5

()

95% ii

.

5.2 Considerations for the conventional approach

The conventional approach is based on the assumption that, according to the design used in ISO 5725-2,

all tests are performed under repeatability conditions in each of the laboratories involved. In particular,

all tests in the laboratory are performed by the same technician, with the same equipment, under the

same conditions and directly one after the other. Test results are considered to have been obtained

from different laboratories under reproducibility conditions, i.e. many factors contribute to observed

variability, e.g. differences in equipment, environmental conditions, reagent batches or technician.

NOTE Validation protocols according to the conventional approach based on LOD and POD can be found

[7]

in ISO 16140-2, ISO 16140-4 and ISO/TS 16393 and AOAC Guidelines . Examples and further protocols are

discussed e.g. in References [8][9][10][11][12] and [13].

5.3 Considerations for the factorial approach

Compared to the conventional approach, in which tests are made under repeatability conditions in each

of the laboratories, the factorial approach systematically varies one or more factors. For instance, half

the tests are conducted with reagents from batch A, and the other half with reagents from batch B.

Thus, the factorial approach makes it possible to ensure the full spectrum of test conditions is covered

in the validation study and assess contributions to variability from separate sources of error. This

approach translates to more efficient and reliable estimation of the total variability.

NOTE Validation protocols based on LOD for microbiological methods according to the factorial approach

[4] [5]

are given in ISO 16140-4 and ISO 16140-5 .

3

© ISO 2023 – All rights reserved

---------------------- Page: 8 ----------------------

ISO/TS 27878:2023(E)

6 Conventional approach

6.1 Experimental design

Results from at least 8 participants, 4 concentration levels, and 8 replicates per level and laboratory

are required to obtain a statistically reliable POD curve. However, with such a design, the reliability

of the results may not be sufficient and will need to be checked. For more reliable estimation of the

LOD and the corresponding variability, it is recommended that results from at least 8 participants,

5 concentration levels, and 12 replicates per level and laboratory are available. If the number of

participants is increased, the number of replicates can be reduced.

The lowest concentration level should be selected so that no further reduction in POD is expected,

even if the concentration level is further reduced. The highest concentration level should be selected in

such a way that no further increase in POD is to be expected even if the concentration level is further

increased. The expected proportions of positive test results across laboratories should be between

20 % and 80 % for at least two concentration levels.

The proportion of positive test results expected at the beginning of the collaborative trial usually differs

from the final POD. This may mean that the proportion of positive test results actually determined in

the collaborative trial does not meet the above requirements. In this case, the results of the evaluation

and, in particular, the calculated reproducibility of the LOD can only be regarded as an estimate.

NOTE These recommendations for the experimental design are based on simulation studies in which the

standard error of the estimate of the laboratory standard deviation was evaluated.

6.2 Statistical model for methods for continuous measurands

The calculation of the LOD is based on a generalized linear mixed-effects model (GLMM) together with

a four-parameter sigmoid curve given by Formula (1):

LH−

POD=P = +H (1)

ii

B

x

1+

aC

i

where

i denotes the laboratory (i = 1, 2,., p);

POD = P denote the probability of detection for laboratory i;

i i

x denotes a given concentration level;

L, H, B, C are global model parameters (i.e. they are valid across all laboratories);

a denotes the laboratory-specific correction of laboratory i;

i

C denotes the global inflection point C.

It is assumed that the parameters, L (lowest probability of detection), H (highest probability of

detection), and B (slope) are the same for all laboratories. The product a C describes the location of the

i

inflection point of the curve for laboratory i; for L = 0 %, H = 100 %, it corresponds to the concentration

at which a POD of 50 % is reached. The value of this product is thus a direct measure of the performance

of the specific laboratory. The parameter, C, corresponds to the performance of an average laboratory.

4

© ISO 2023 – All rights reserved

---------------------- Page: 9 ----------------------

ISO/TS 27878:2023(E)

The a values are modelled as realizations of a random variable: It is assumed that the ln a values follow

i i

a normal distribution with

2

lna ∼N 0,σ

()

i L

2

The parameters L, H, B, C and σ can be provided by maximum likelihood estimation, e.g. in

L

2

mathematical-statistical software package. The variance σ characterizes the variability of sensitivity

L

between laboratories.

NOTE 1 Although there is no guarantee that the distribution of ln a values actually follows a normal

i

distribution, the log transformation usually leads to a better approximation of the normal distribution. If the

method displays poor precision, then the prediction range of the LOD values without log transformation could

include infeasible negative values.

NOTE 2 It is assumed that the parameters L, H, C and B are the same for all laboratories, i.e. that the shape

of the curve is sigmoidal and the same across laboratories. It should be checked whether this assumption is

justified, e.g. through a graphic check of laboratory-specific POD curves.

The interpretation of the parameters will be explained with an example, see Reference [13]. A

collaborative study of a method for the binary analysis of gluten in corn products was conducted to

demonstrate that the binary test method can detect gluten contaminations below the threshold of

20 mg/kg gluten. A total of four corn sample lots with different gluten concentrations was submitted

to 18 laboratories to evaluate the sensitivity and reproducibility of the test method. Each of the 18

laboratories conducted 10 tests for each of four concentration levels. Table 1 provides the corresponding

numbers of positive results per laboratory and concentration level.

Table 1 — Number of positive test results per concentration level and laboratory (10 replicates)

Concentration level

Laboratory

No.

0,88 mg/kg 2,42 mg/kg 5,48 mg/kg 9,38 mg/kg

01 0 10 10 10

02 0 10 10 10

03 0 10 10 10

04 0 10 10 10

05 0 10 10 10

06 0 10 10 10

07 0 10 10 10

08 0 9 10 10

09 0 10 10 10

10 0 9 8 10

11 0 10 10 10

12 0 10 10 10

13 0 10 10 10

14 0 10 10 10

15 0 9 10 10

16 0 10 10 10

17 0 10 10 10

18 2 10 10 10

Figure 1 shows the POD curve of a laboratory with average performance (solid line) along with 95 %

prediction range of laboratory-specific POD (dark grey zone) and 95 % prediction range of laboratory-

specific RODs (light grey step-functions). The numbers adjacent to the diamonds indicate the

laboratory numbers having obtained the corresponding ROD.

5

© ISO 2023 – All rights reserved

---------------------- Page: 10 ----------------------

ISO/TS 27878:2023(E)

For instance, at the concentration level 0,88 mg/kg, one laboratory has an ROD of 0,2, and 17

laboratories have an ROD of 0. Comparison with Table 1 shows that the laboratory with the ROD of

0,2 is laboratory 18. The light grey step-functions show the 95 % prediction range for the ROD values,

obtained from simulation runs performed on the basis of the parameter estimates (Monte Carlo

simulation). Figure 1 can be read as follows: a POD of 80 % is reached by a laboratory with an average

performance at a concentration of about 1,7 mg/kg (solid line), whereas a top-performing laboratory

will reach this POD at 1,3 mg/kg (upper dark grey zone) and a low-performing laboratory will need a

concentration of about 2,2 mg/kg (lower dark grey zone).

NOTE 3 None of the selected concentration levels is within the 20 % to 80 % interval; therefore, the calculated

reproducibility data can only be considered as an inaccurate estimate.

Key

X concentration, in mg/kg Y POD and ROD

Figure 1 — Mean POD curve, laboratory-specific RODs and prediction ranges

A special case of the model in 6.2 with L = 0, H = 1 and constant a value is equivalent to the logit

i

model for x > 0. In other words, the logit model is already included in the model in 6.2. In practical

terms, this statement also holds for the probit model, since it is very similar to the logit model, see e.g.

Reference [14].

If continuous test results are available, the validation study should be based on these rather than on

the corresponding binary results. In other words, insofar as binary results are obtained by comparing

continuous test results to a threshold, the laboratories should submit the original continuous results,

and the comparison with the threshold should be conducted as part of the validation study.

In many cases, the original continuous results will not be available, of course. In particular, in many

cases, the assay yields a binary result, even though it is based on a continuous response.

2

Finally, it should be noted that the estimate of the between-laboratory variance σ obtained from the

L

binary results on the basis of the model described above is closely related to the between-laboratory

standard deviation σ from ISO 5725-2. Indeed, if p laboratories each submitted two replicate LOD

L

6

© ISO 2023 – All rights reserved

---------------------- Page: 11 ----------------------

ISO/TS 27878:2023(E)

values in a collaborative study, it would be possible to consider the σ estimate computed according to

L

ISO 5725-2 to be equivalent to the σ estimate as computed here.

L

2

NOTE 4 Given an estimate for a variance (such as the between-laboratory variance estimate σ mentioned

L

above), the corresponding standard deviation is obtained by taking the square root.

6.3 Statistical model for methods for discrete measurands

In the case of measurands quantified on a discrete scale (e.g. microbiological culture methods or

PCR methods), the four-parameter model discussed in 6.2 is no longer appropriate. The reason is the

difference in distributional assumptions regarding the concentration, x. In 6.2, x denotes a nominal

concentration level per se, and differences between the actual concentration of a test portion and

the nominal concentration level can be assumed to be negligible. In the case of discrete measurands,

x denotes e.g. the number of colony-forming units or DNA copies per test portion. For the sake of

terminological convenience, these discrete quantities are referred to as concentration levels (see Note 1

to entry to definition 3.3) but, in the case of the discrete measurands considered here, differences

between the actual concentration in a test portion and the nominal concentration can no longer be

assumed to be negligible; rather, for a given nominal concentration level, the actual concentration levels

of test portions are assumed to be subject to random variability and to follow a Poisson distribution.

This assumption will be referred to in the following as the “Poisson assumption”. For this reason, the

cloglog (complementary-log-log) model is appropriate for the calculation of the LOD and its variability

in the case of discrete measurands; accordingly, the following generalized mixed linear model (GLMM)

is applied:

ln{}−−ln[]1POD ()xa=+ln bxln

i i

where

i denotes the laboratory (i = 1,2,.,p);

x denotes a given concentration level;

b is a global positive parameter that models the dependence of the sensitivity on the concentration

level;

a denotes the sensitivity corresponding to laboratory i.

i

NOTE 1 The Poisson assumption requires that POD ()xP==0 for x= 0 . This means that the above model

ii

should only be used if the number of false-positive results is negligible. Another consequence of the Poisson

assumption is that POD ()xP= ()x will approach 1 with increasing x; in other words, the model is also

ii

susceptible to false negatives. This assumption constitutes an important difference to the four-parameter model

discussed in 6.2, which admits both false positives and false negatives.

NOTE 2 The complementary log-log model is a standard model for microbiological methods and qualitative

PCR. The model establishes a relationship between the probability of a positive result and the concentration,

when a test portion is taken from a homogeneous sample. It is assumed that the probability of detecting an

individual cell or DNA (RNA) copy does not depend on the concentration level. The probability of a positive result

is then simply derived from the Poisson distribution: a qualitative result is positive if at least one cell or DNA

(RNA) copy is detected.

It is assumed that the ln a values follow a normal distribution with

i

2

lnaμ∼N μσ,, = lna.

()

i L

2

The three parameters a, σ and b can be determined by maximum likelihood estimation in standard

L

statistical software such as R. The parameter a represents the average sensitivity parameter (at x

**...**

## Questions, Comments and Discussion

## Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.