ISO/IEC 19795-10:2024
(Main)Information technology — Biometric performance testing and reporting — Part 10: Quantifying biometric system performance variation across demographic groups
Information technology — Biometric performance testing and reporting — Part 10: Quantifying biometric system performance variation across demographic groups
This document establishes requirements for estimating and reporting on performance variations observed when cohorts belonging to different demographic groups engage with biometric enrolment and recognition systems. In this context, performance refers to failure-to-enrol rate, failure-to-acquire rate, shifts in comparison score, recognition error rates, and aspects of response and processing time (throughput). This document is applicable to the following: — demographic group membership; — using phenotypic measures; — reporting on tests; — stating statistical uncertainty estimates; — operational thresholds settings; — equitability; — procurement agency activities. This document also provides terms and definitions to be used when reporting performance variation across demographic groups. This document is applicable to: — technology evaluations of algorithms, subsystems and systems; — scenario evaluations of systems; — operational evaluations of fielded systems. Application of this document does not require detailed knowledge of a system’s algorithms but it does require specific knowledge of the demographic characteristics for the population of interest.
Technologies de l'information — Essais et rapports de performance biométriques — Partie 10: Quantification de la variation des performances du système biométrique selon les groupes démographiques
General Information
Standards Content (Sample)
International
Standard
ISO/IEC 19795-10
First edition
Information technology —
2024-10
Biometric performance testing and
reporting —
Part 10:
Quantifying biometric system
performance variation across
demographic groups
Technologies de l'information — Essais et rapports de
performance biométriques —
Partie 10: Quantification de la variation des performances du
système biométrique selon les groupes démographiques
Reference number
© ISO/IEC 2024
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2024 – All rights reserved
ii
Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 2
4 Conformance . 4
5 Planning the evaluation . 4
5.1 Identifying the scope of the evaluation .4
5.2 Demographic variables .5
5.2.1 Ground truth requirements .5
5.2.2 Categorical demographic variables .5
5.2.3 Continuous demographic variables .7
5.2.4 Other demographic variables . .8
6 Executing the evaluation . 8
6.1 Generation of mated comparison and identification trials .8
6.2 Generation of non-mated comparison and identification trials .8
6.2.1 General .8
6.2.2 Verification (1:1) .8
6.2.3 Identification (1:N) .8
6.3 Selection of a threshold .9
6.4 Calculating differential performance based on categorical variables for two specific
demographic groups .9
6.4.1 General .9
6.4.2 Differential performance between two groups based on mathematical difference.9
6.4.3 Differential performance between two groups based on mathematical ratio .10
6.5 Calculating differential performance based on categorical variables for more than two
groups .10
6.5.1 General .10
6.5.2 Differential performance for more than two groups based on the largest error
rate relative to the geometric mean .10
6.5.3 Differential performance for more than two groups based on the Gini coefficient .11
6.6 Calculating differential performance in identification trials .11
6.7 Calculating demographic differentials for failure-to-enrol rate, failure-to-acquire rate
and transaction duration . 12
6.8 Calculating demographic differentials for continuous variables . 12
6.9 Comparison score differential measures . 13
6.10 Calculating uncertainty .14
6.10.1 Uncertainty in demographic differentials .14
6.10.2 Sampling the target population .14
6.10.3 Sample size requirements . 15
7 Reporting the evaluation results .16
7.1 Reporting the experimental design .16
7.2 Reporting the target application .16
7.3 Reporting the test population .16
7.4 Reporting differential performance .17
7.4.1 Reporting differential performance on previously collected datasets. .17
7.4.2 Reporting differential performance for two or more groups .17
7.4.3 Reporting differential performance against a benchmark .18
7.4.4 Reporting error trade-off metrics .18
7.4.5 Reporting threshold management policy .18
7.5 Reporting comparison score differential measures .18
7.6 Reporting exception handling .19
© ISO/IEC 2024 – All rights reserved
iii
Annex A (informative) Example of estimating sample size for differential performance .20
Annex B (informative) Calculating aggregate equitability measures .23
Bibliography .25
© ISO/IEC 2024 – All rights reserved
iv
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical activity.
ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations,
governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types
of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/
IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any
claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held
responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html.
In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 37, Biometrics.
A list of all parts in the ISO/IEC 19795 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.
© ISO/IEC 2024 – All rights reserved
v
Introduction
As the use of biometric technology increases, so too does public interest in establishing whether the
technology performs similarly for all individuals. Stakeholders are asking government and industry
organizations that use biometric technology to establish whether these technologies vary in performance
for different demographic groups. The intention of this document is to provide guidance on how to measure
[2]
and report performance variation across demographic groups.
This document is intended to help organizations evaluate demographic performance in biometric systems
and report their results. Specifically, this document outlines how to measure and report biometric
performance variations across demographic groups. It provides a set of metrics and best practices to
facilitate such testing. However, this document does not provide guidance on how to establish specific
causes for the observed variations. The fol
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.