ISO/IEC 19795-6:2012

Information technology — Biometric performance testing and reporting — Part 6: Testing methodologies for operational evaluation

ISO/IEC 19795-6:2012: provides guidance on the operational testing of biometric systems; specifies performance metrics for operational systems; details data that may be retained by operational systems to enable performance monitoring; and specifies requirements on test methods, recording of data, and reporting of results of operational evaluations. ISO/IEC 19795-6:2012 does not: cover testing of operational systems in the laboratory or address vulnerability testing.

Technologies de l'information — Essais et rapports de performances biométriques — Partie 6: Méthodologies d'essai pour l'évaluation opérationnelle

General Information

• Status: Published
• Publication Date: 22-Jan-2012
• ICS: 35.040 - Information coding
• ICS: 35.240.15 - Identification cards. Chip cards. Biometrics
• Technical Committee: ISO/IEC JTC 1/SC 37 - Biometrics
• Drafting Committee: ISO/IEC JTC 1/SC 37/WG 5 - Biometric testing and reporting
• Current Stage: 9060 - Close of review
• Start Date: 02-Sep-2028

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 19795-6
First edition
2012-02-01

Information technology — Biometric
performance testing and reporting —
Part 6:
Testing methodologies for operational
evaluation
Technologies de l'information — Essais et rapports de performances
biométriques —
Partie 6: Méthodologies d'essai pour l'évaluation opérationnelle




Reference number
ISO/IEC 19795-6:2012(E)
©
ISO/IEC 2012

---------------------- Page: 1 ----------------------
ISO/IEC 19795-6:2012(E)

COPYRIGHT PROTECTED DOCUMENT


©  ISO/IEC 2012
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56  CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail [e-mail]
Web www.iso.org
Published in Switzerland

ii © ISO/IEC 2012 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC 19795-6:2012(E)
Contents Page
Foreword . v
Introduction . vi
1  Scope . 1
2  Conformance . 1
3  Normative references . 1
4  Terms and definitions . 2
5  Operational evaluation overview . 3
5.1  Operational evaluation goals . 3
5.2  Operational performance metrics . 4
5.3  Operational evaluation methods . 4
5.4  Determining operational performance . 4
5.5  Use of technology and scenario evaluation methodologies in evaluating operational
systems . 5
6  Operational evaluation . 5
6.1  Purpose and scope . 5
6.1.1  General . 5
6.1.2  Criteria for system inclusion . 5
6.1.3  System specification . 5
6.1.4  Biometric functionality . 6
6.1.5  Performance measures . 6
6.2  Application characteristics . 6
6.2.1  General . 6
6.2.2  Concept of operations . 7
6.2.3  Guidance and instruction . 7
6.2.4  Levels of effort and decision policies . 8
6.2.5  Multiple-instance systems . 8
6.2.6  Environment . 9
6.2.7  Deployment factors . 9
6.2.8  Acclimatization . 10
6.2.9  Habituation . 10
6.3  Test Plan . 10
6.3.1  General . 10
6.3.2  System implementation and configuration. 11
6.3.3  Test population . 11
6.3.4  Test transactions . 12
6.4  Performance measurement . 14
6.4.1  Throughput . 14
6.4.2  Enrolment analysis . 15
6.4.3  Recognition analysis . 15
6.5  Reporting . 16
6.5.1  Reporting planned test results . 16
6.5.2  Reporting additional analyses . 16
6.5.3  Reporting observations . 17
6.5.4  Report structure . 17
6.6  Record keeping . 17
Annex A (informative) Non-mandatory performance metrics and reporting . 18
Annex B (informative) Sub-transaction events in operational testing . 20
© ISO/IEC 2012 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC 19795-6:2012(E)
Annex C (informative) Sample operational test specification .21
Annex D (informative) Methods to determine test size .23
Annex E (informative) Operational system monitoring .25
Annex F (informative) Operational habituation testing .27
Annex G (informative) Sample operational test report outline .28
Bibliography .30

iv © ISO/IEC 2012 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC 19795-6:2012(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 19795-6 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 37, Biometrics.
ISO/IEC 19795 consists of the following parts, under the general title Information technology — Biometric
performance testing and reporting:
 Part 1: Principles and framework
 Part 2: Testing methodologies for technology and scenario evaluation
 Part 3: Modality-specific testing [Technical Report]
 Part 4: Interoperability performance testing
 Part 5: Access control scenario and grading scheme
 Part 6: Testing methodologies for operational evaluation
 Part 7: Testing of on-card biometric comparison algorithms
© ISO/IEC 2012 – All rights reserved v

---------------------- Page: 5 ----------------------
ISO/IEC 19795-6:2012(E)
Introduction
Operational tests evaluate complete biometric systems in the targeted operational environment with the target
population. Tests may encompass performance monitoring of operational systems or assessment of
performance in operational trials.
Operational performance assessment may be based on:
 data collected by the operational system in the course of normal operation;
 additional data collected during normal system use, but with the system running in an “evaluation mode”
allowing extra data to be collected;
 data collected with a set of test subjects considered separately from the subject base of the operational
system.
Operational evaluation differs from technology or scenario evaluation in that the subject base, environment,
and system design are no longer controlled for the purpose of repeatable testing, but vary in accordance with
operational use. Examples of uncontrolled variables include the legitimacy of the subject’s identity claim,
environmental effects from weather or lighting, or the variability of system use by different individuals.
The overarching goals of operational testing are to measure or monitor operational biometric system
performance over a period of time.
Subgoals of operational testing may include:
 to determine if performance meets the requirements specified for a particular application or the claims
asserted by the supplier;
 to determine the need to adjust or configure the system to improve performance;
 to predict performance as the numbers of subjects, locations, or devices increase;
 to obtain information on the target population and environmental parameters found to affect system
performance;
 to obtain performance data from a pilot implementation;
 to obtain performance data to benchmark future systems.
This part of ISO/IEC 19795 provides the test planning, test conduct, performance measurement, test reporting,
and record keeping requirements to be followed during a biometric system’s operational evaluation.
vi © ISO/IEC 2012 – All rights reserved

---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 19795-6:2012(E)

Information technology — Biometric performance testing and
reporting —
Part 6:
Testing methodologies for operational evaluation
1 Scope
This part of ISO/IEC 19795:
 provides guidance on the operational testing of biometric systems;
 specifies performance metrics for operational systems;
 details data that may be retained by operational systems to enable performance monitoring; and
 specifies requirements on test methods, recording of data, and reporting of results of operational
evaluations.
NOTE Some operational biometric systems perform a single biometric function. For example, in the initial stages of
rollout of biometric passports, the operational system might be performing biometric enrolment only. Operational
evaluation of such systems is within the scope of this part of ISO/IEC 19795.
This part of ISO/IEC 19795 does not:
 cover testing of operational systems in the laboratory or
 address vulnerability testing.
2 Conformance
An operational evaluation is in conformance with this part of ISO/IEC 19795 if it is planned, executed and
reported in accordance with the requirements of Clause 6.
3 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced
document (including any amendments) applies.
ISO/IEC 19795-1, Information technology — Biometric performance testing and reporting — Part 1: Principles
and framework
© ISO/IEC 2012 – All rights reserved 1

---------------------- Page: 7 ----------------------
ISO/IEC 19795-6:2012(E)
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 19795-1 and the following apply.
4.1
acclimatization
change, over the course of one or more transactions, of a biometric characteristic that might impact the ability
of a system to process a sample
NOTE Acclimatization is primarily associated with a subject’s temporal adjustment to environmental effects, such as
skin temperature.
4.2
attendant
agent of the biometric system operator who directly interacts with the biometric capture subject
4.3
biometric capture subject
individual who is the subject of a biometric capture process
4.4
biometric data subject
individual whose individualized biometric data is within the biometric system
4.5
biometric probe
biometric data input to an algorithm for comparison to a biometric reference(s)
4.6
biometric operational personnel
individuals, other than the biometric capture subjects, who take an active role in the operation of the biometric
system
NOTE Biometric operational personnel includes biometric system administrators, attendants, and examiners.
4.7
biometric system administrator
person who executes policies and procedures in the administration of a biometric system
4.8
biometric system operator
organization responsible for defining policies and procedures in the operation of a biometric system
4.9
biometric reference
one or more stored biometric samples, biometric templates or biometric models attributed to a biometric data
subject and used for comparison
4.10
comparison attempt limit
maximum allowed number or duration of attempts in a comparison transaction
4.11
enrolment attempt limit
maximum allowed number or duration of attempts in an enrolment transaction
4.12
habituation
familiarity a subject has with the biometric device, system and application
NOTE The level of habituation can affect biometric sample presentation and acquisition device.
2 © ISO/IEC 2012 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC 19795-6:2012(E)
4.13
subject base
set of individuals whose biometric data is intended to be enrolled or compared in operational use of a
biometric system
4.14
system acceptance rate
proportion of recognition transactions in an operational system in which the subject is recognized
NOTE 1 Though the acceptance of an impostor is an incorrect recognition, it can still count as a system acceptance.
NOTE 2 System acceptance rate = 1 – system rejection rate.
4.15
system identification rate
proportion of identification transactions in an operational system in which one or more subjects are identified
4.16
system rejection rate
proportion of recognition transactions in an operational system in which the subject is not recognized
NOTE The system rejection rate differs from the false reject rate in that, in addition to false rejections, it also includes
any rejected impostor transaction and any improper genuine transaction.
4.17
test crew member
selected biometric data subject whose use of the operational system is controlled or monitored as part of the
evaluation
NOTE In an operational evaluation, test subjects can be subjects of the operational system or they can be members
of a test crew using the system specifically for evaluation purposes
5 Operational evaluation overview
5.1 Operational evaluation goals
The overarching goals of operational testing are to measure or monitor operational biometric system
performance.
Subgoals of operational testing may include:
 to determine if performance meets the requirements specified for a particular application or the claims
asserted by the supplier;
 to determine the need to adjust or configure the system to improve performance;
 to predict performance as the numbers of subjects, locations, or devices increase;
 to obtain information on the target population and environmental parameters found to affect system
performance;
 to obtain performance data from a pilot implementation;
 to obtain performance data to benchmark future systems.
Operational evaluation considers the performance of people as well as the equipment, algorithms, and
environment. Consequently, operational testing includes aspects of social science in addition to physical
science, whereas technology testing does not. In general, operational performance will vary over time due to
uncontrolled conditions in people, equipment and environment. For example, if the majority of subjects are
enrolled at the start of operations, with few new enrolments, performance of the system might improve as
© ISO/IEC 2012 – All rights reserved 3

---------------------- Page: 9 ----------------------
ISO/IEC 19795-6:2012(E)
subjects habituate or degrade as subjects’ biometric characteristics age over time away from their enrolled
references.
The performance observed in testing can depend on the operational personnel, such as attendants or
biometric examiners, as well as the biometric subjects. Operational personnel based factors should be taken
into consideration in all aspects of the test from scope definition to reporting (see e.g., references [1] and [2]).
5.2 Operational performance metrics
Recognition metrics for operational testing differ from those used in technology and scenario testing. In
technology and scenario tests, false accept and false reject rates can be measured because the underlying
ground truth is known to the experimenters. Ground truth will generally be unknown in an operational setting
such that an operational test will measure system acceptance and system rejection rates.
Determining the false accept rate and false reject rate from the number of system rejections and acceptances
will require additional observations or controls to determine the legitimacy of identity claims and device
interactions. Similarly, in the case of identification systems, determining identification error rates from the
number of system identifications also requires additional observations or controls.
Performance calculations in technology and scenario tests often exclude rejections in which the subject did
not provide an ideal presentation or did not correctly follow instructions. Operational testing will include such
rejections in measuring the system rejection rate.
5.3 Operational evaluation methods
Operational performance assessment is based on data collected through an operational system. A system
may be configured in “evaluation mode” to collect additional data during normal system use.
Operational performance assessment may be based upon different groups of test subjects:
 data collected with a non-controlled set of test subjects (i.e., a set of test subjects reflective of the subject
base of the operational system);
 data collected with a controlled set of test subjects defined as the “test crew members” (i.e., a set of test
subjects considered and controlled separately from the subject base of the operational system).
If the test crew is specially instructed in their use of the system, evaluation results can be expected to differ
from those encountered operationally. Operational performance can be highly time variant because of
uncontrollable variations in the population, equipment and environment. Variation in performance across these
conditions cannot be predicted.
5.4 Determining operational performance
Performance estimates may be determined from operational data in at least three ways:
 through direct observation of throughput rates, acceptance rates and/or rejection rates;
 through offline computation of throughput and acceptance and rejection rates based on comparison
scores and timing metrics recorded during operations; and
 through offline computation of comparison scores, and acceptance and rejection rates based on samples
acquired during the test and on stored reference data.
Each approach is likely to yield different measures. Further, each approach carries different risks for
miscalculation. For example, recording only direct observations of an access control system does not reveal
whether a rejection was due to a biometric error or an error in the operation of the gate; recording only
comparison scores will not show cases where a reference or probe has been stored or transmitted incorrectly.
4 © ISO/IEC 2012 – All rights reserved

---------------------- Page: 10 ----------------------
ISO/IEC 19795-6:2012(E)
5.5 Use of technology and scenario evaluation methodologies in evaluating operational
systems
In addition to tests based on real operational use of the system, scenario and technology evaluation
(ISO/IEC 19795-2 [3]) can also have a role in determining some aspects of operational performance.
Testing solely in live operation might not be capable of measuring all aspects of operational performance.
Depending on the purpose of the evaluation, certain performance measures might only be determined by
testing for them specifically. Testing in live operation is not meant to guarantee that the system will be
operating under the specific conditions, or with sufficient frequency, in order to draw statistically valid
conclusions. Furthermore, when testing operationally, it might be infeasible to isolate these effects from other
operational factors that also affect performance.
EXAMPLE Environmental factors such as sunlight or humidity can affect sensor performance. Unless the system is
tested to monitor performance in the specific environments, the effects that such factors have on the operational
performance cannot be quantified.
6 Operational evaluation
6.1 Purpose and scope
6.1.1 General
The purpose and scope of the test need to be determined before the test design can be drawn up. The
following elements shall be addressed:
 criteria for system inclusion,
 system specification,
 enrolment and comparison functionality to be evaluated, and
 performance measures of interest.
6.1.2 Criteria for system inclusion
The experimenter shall address the criteria by which biometric system(s) are included in an operational test.
Biometric systems may be included in an operational test due to their having been previously deployed, due to
selection on the part of the biometric system operator, or due to selection on the part of the evaluating entity.
NOTE Testing multiple independent systems could compromise the operational realism of the tests. However, some
elements of the test must be controlled if meaningful comparisons are to be made. Some elements can be controlled
without jeopardizing the operational value of the tests. (See reference [4].)
6.1.3 System specification
Details of the system under test shall be specified as fully as possible. The following elements should be
reported:
 for acquisition devices: manufacturer, model, version, and firmware version as applicable — if the
acquisition device’s core acquisition components are integrated within a third-party device, such as in the
case of a fingerprint sensor incorporated into a peripheral, then manufacturer, model, version, and
firmware of the core acquisition components and those of the peripheral shall be reported;
 for biometric algorithms: provider, version, and revisions, and the values of all field-variable parameters or
settings — biometric algorithms include quality assessment, feature extraction, binning, comparison and
fusion algorithms, and any or all of these might be supplied by different vendors;
© ISO/IEC 2012 – All rights reserved 5

---------------------- Page: 11 ----------------------
ISO/IEC 19795-6:2012(E)
 if the operational system incorporates a biometric application, such as a logical access interface: provider,
title, version, and build of the application;
 for systems tested on or through personal computers, personal digital assistants or other computing
devices: platform, operating systems, processing power, memory, manufacturer, and model of computing
device;
 details of system architecture and data flow between biometric data acquisition, processing, and storage
components;
 data flow between system components;
 system configuration (e.g., in the case reference updating, does the system use a single biometric
reference for all subsequent comparison attempts or is the biometric reference updated following each
successful attempt).
6.1.4 Biometric functionality
In operational tests of previously deployed systems, the biometric functionality (i.e., enrolment, verification or
identification) under evaluation should be that of the deployed system. In operational tests of systems
deployed for the purpose of operational testing, the experimenter may determine which comparison
functionality(ies) to implement and evaluate. The rationale for selecting the comparison functionality
components to be evaluated by the operational test shall be reported.
NOTE An operational test can incorporate both identification and verification functions if, for example, data is used to
execute searches against watch lists and also for verification against an existing enrolment.
6.1.5 Performance measures
Performance measures relevant for operational evaluation include:
 throughput for enrolment and recognition transactions,
 failure-to-enrol rate,
 system rejection rate (in verification systems),
 system identification rate (in identification systems)
 false accept rate and false reject rate (in verification systems when the evaluation can establish ground
truth),
 false-positive identification error rate and false-negative identification error rate (in identification systems
when the evaluation can establish ground truth).
Considerations for determining and reporting these performance measures are specified in Clause 6.4. In
addition, experimenters shall determine any specific performance measures to be generated through the
operational test. Annex A provides a list of potential performance metrics.
6.2 Application characteristics
6.2.1 General
Application characteristics shall be considered in order to plan for test data collection that will be
representative of operational use. The following elements shall be addressed:
 concept of operations,
 guidance and instruction,
6 © ISO/IEC 2012 – All rights reserved

---------------------- Page: 12 ----------------------
ISO/IEC 19795-6:2012(E)
 levels of effort and decision policies,
 use of multiple instances,
 environment,
 deploym
...