Information technology -- Biometric performance testing and reporting

Technologies de l'information -- Essais et rapports de performance biométriques

General Information

Status
Published
Publication Date
26-May-2021
Current Stage
5060 - Close of voting Proof returned by Secretariat
Start Date
14-Apr-2021
Completion Date
13-Apr-2021
Ref Project

Buy Standard

Draft
ISO/IEC FDIS 19795-1 - Information technology -- Biometric performance testing and reporting
English language
78 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

FINAL
INTERNATIONAL ISO/IEC
DRAFT
STANDARD FDIS
19795-1
ISO/IEC JTC 1/SC 37
Information technology — Biometric
Secretariat: ANSI
performance testing and reporting —
Voting begins on:
2021-02-16
Part 1:
Voting terminates on:
Principles and framework
2021-04-13
Technologies de l'information — Essais et rapports de performance
biométriques —
Partie 1: Principes et canevas
RECIPIENTS OF THIS DRAFT ARE INVITED TO
SUBMIT, WITH THEIR COMMENTS, NOTIFICATION
OF ANY RELEVANT PATENT RIGHTS OF WHICH
THEY ARE AWARE AND TO PROVIDE SUPPOR TING
DOCUMENTATION.
IN ADDITION TO THEIR EVALUATION AS
Reference number
BEING ACCEPTABLE FOR INDUSTRIAL, TECHNO-
ISO/IEC FDIS 19795-1:2021(E)
LOGICAL, COMMERCIAL AND USER PURPOSES,
DRAFT INTERNATIONAL STANDARDS MAY ON
OCCASION HAVE TO BE CONSIDERED IN THE
LIGHT OF THEIR POTENTIAL TO BECOME STAN-
DARDS TO WHICH REFERENCE MAY BE MADE IN
NATIONAL REGULATIONS. ISO/IEC 2021
---------------------- Page: 1 ----------------------
ISO/IEC FDIS 19795-1:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC FDIS 19795-1:2021(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................vi

Introduction ..............................................................................................................................................................................................................................vii

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Abbreviated terms .............................................................................................................................................................................................. 5

5 Conformance ............................................................................................................................................................................................................. 6

6 General biometric system ........................................................................................................................................................................... 6

6.1 Conceptual representation of general biometric system ................................................................................... 6

6.2 Conceptual components of a general biometric system...................................................................................... 7

6.2.1 Data capture subsystem ............................................................................................................................................ 7

6.2.2 Transmission subsystem........................................................................................................................................... 7

6.2.3 Signal processing subsystem ................................................................................................................................. 7

6.2.4 Data storage subsystem ............................................................................................................................................. 7

6.2.5 Comparison subsystem .............................................................................................................................................. 7

6.2.6 Decision subsystem ....................................................................................................................................................... 8

6.2.7 Administration subsystem ...................................................................................................................................... 8

6.2.8 Interface to external application ........................................................................................................................ 8

6.3 Functions of general biometric system .............................................................................................................................. 9

6.3.1 Enrolment .............................................................................................................................................................................. 9

6.3.2 Verification of a positive biometric claim ................................................................................................... 9

6.3.3 Identification ....................................................................................................................................................................10

6.4 Enrolment, verification and identification transactions ..................................................................................11

6.5 Performance measures .................................................................................................................................................................12

6.5.1 Error rates ..........................................................................................................................................................................12

6.5.2 Throughput rates ..........................................................................................................................................................12

6.5.3 Types of performance testing ............................................................................................................................13

7 Planning the evaluation .............................................................................................................................................................................13

7.1 General ........................................................................................................................................................................................................13

7.2 Determine information about the system ....................................................................................................................14

7.3 Controlling factors that influence performance ......................................................................................................15

7.4 Test subject selection......................................................................................................................................................................16

7.5 Test size ......................................................................................................................................................................................................17

7.5.1 General...................................................................................................................................................................................17

7.5.2 Collecting multiple recognition transactions per test subject per system ..................17

7.5.3 Requirements on test size .....................................................................................................................................18

7.6 Multiple tests .........................................................................................................................................................................................18

8 Data collection .....................................................................................................................................................................................................19

8.1 Avoidance of data collection errors ...................................................................................................................................19

8.2 Data and details collected ...........................................................................................................................................................19

8.3 Enrolments ..............................................................................................................................................................................................20

8.3.1 Enrolment transactions ..........................................................................................................................................20

8.3.2 Enrolment conditions ...............................................................................................................................................21

8.3.3 Enrolment failures and presentation errors .........................................................................................21

8.4 One-to-one comparison trials .................................................................................................................................................22

8.4.1 General...................................................................................................................................................................................22

8.4.2 Collection conditions ........................................................................................................................................... ......22

8.4.3 Frequency of use ...........................................................................................................................................................22

8.4.4 Systems performing optimization based on enrolled references .......................................23

8.4.5 Systems performing reference adaptation .............................................................................................23

8.4.6 Processes for data entry errors and system misuse ......................................................................23

© ISO/IEC 2021 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC FDIS 19795-1:2021(E)

8.4.7 Failures to acquire .......................................................................................................................................................23

8.4.8 Adding test data to the corpus ..........................................................................................................................23

8.4.9 Online comparison trials .......................................................................................................................................23

8.4.10 Offline comparison trials .......................................................................................................................................24

8.4.11 Offline non-mated comparison trials when references are dependent .................. .......25

8.4.12 Offline non-mated comparison trials based on comparison of references ................25

8.4.13 Use of samples from multi-capture comparison transactions ..............................................25

8.5 Identification trials ...........................................................................................................................................................................26

8.5.1 General...................................................................................................................................................................................26

8.5.2 Identification testing with non-enrolled test subjects .................................................................26

8.5.3 Use of jack-knife approach for identification testing .....................................................................26

9 Analyses ......................................................................................................................................................................................................................26

9.1 General ........................................................................................................................................................................................................26

9.2 Performance of biometric enrolment ...............................................................................................................................27

9.2.1 Failure-to-enrol rate...................................................................................................................................................27

9.2.2 Enrolment transaction duration......................................................................................................................27

9.3 Performance of biometric acquisition .............................................................................................................................28

9.3.1 Failure-to-acquire rate .............................................................................................................................................28

9.3.2 Acquisition process duration .............................................................................................................................28

9.3.3 Other aspects of acquisition performance ..............................................................................................28

9.4 One-to-one comparison performance..............................................................................................................................29

9.4.1 False non-match rate .................................................................................................................................................29

9.4.2 False match rate .............................................................................................................................................................29

9.5 Verification system performance metrics .....................................................................................................................30

9.5.1 General...................................................................................................................................................................................30

9.5.2 False reject rate ..............................................................................................................................................................30

9.5.3 False accept rate ............................................................................................................................................................31

9.5.4 Verification transaction duration ...................................................................................................................31

9.5.5 Generalized false reject rate and generalized false accept rate ............................................31

9.6 Identification system performance metrics ................................................................................................................32

9.6.1 General...................................................................................................................................................................................32

9.6.2 False-negative identification rate ...................................................................................................................33

9.6.3 False-positive identification rate ....................................................................................................................33

9.6.4 Generalized false-negative identification rate and generalized false-

positive identification rate ...................................................................................................................................34

9.6.5 Selectivity ............................................................................................................................................................................34

9.6.6 Closed-set test of identification performance .....................................................................................35

9.6.7 Estimation of identification error rates from one-to-one comparison results .......35

9.6.8 Predicting identification error rates in larger populations .....................................................36

9.7 Analysis of performance across controlled experimental factors ...........................................................36

9.7.1 Longitudinal analyses ...............................................................................................................................................36

9.7.2 Pairwise analyses .........................................................................................................................................................36

9.8 Detection error trade-off .............................................................................................................................................................36

9.9 Transaction durations ....................................................................................................................................................................37

9.10 Computational workload ............................................................................................................................................................37

9.11 Uncertainty of estimates..............................................................................................................................................................38

10 Graphical presentation of results ....................................................................................................................................................39

10.1 Score distributions ...........................................................................................................................................................................39

10.1.1 General...................................................................................................................................................................................39

10.1.2 Boxplots ................................................................................................................................................................................39

10.2 Error rate vs threshold plot ......................................................................................................................................................40

10.3 DET plot .....................................................................................................................................................................................................40

10.4 CMC plot / FNIR over rank plot .............................................................................................................................................42

10.5 FNIR over number of enrolees plot ....................................................................................................................................44

10.6 Heat maps .................................................................................................................................................................................................45

11 Record keeping ....................................................................................................................................................................................................46

iv © ISO/IEC 2021 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC FDIS 19795-1:2021(E)

12 Reporting performance results ..........................................................................................................................................................47

12.1 Reporting test details .....................................................................................................................................................................47

12.2 Summary statistics ...........................................................................................................................................................................48

12.3 Reporting enrolment performance ....................................................................................................................................48

12.4 Reporting acquisition performance ...................................................................................................................................48

12.5 Reporting one-to-one comparison performance ....................................................................................................49

12.6 Reporting verification system performance ...............................................................................................................49

12.7 Reporting identification system performance ..........................................................................................................49

12.8 Reporting performance across factors ............................................................................................................................50

Annex A (informative) Differences between evaluation types ...............................................................................................52

Annex B (Informative) Test size and random uncertainty ..........................................................................................................53

Annex C (informative) Factors influencing performance .............................................................................................................61

Annex D (Informative) Pre-selection algorithm performance ...............................................................................................66

Annex E (informative) Identification performance as a function of database size ...........................................68

Annex F (informative) Algorithms for generating DET and CMC .........................................................................................70

Annex G (informative) DET properties and interpretation .......................................................................................................73

Bibliography .............................................................................................................................................................................................................................77

© ISO/IEC 2021 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC FDIS 19795-1:2021(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that

are members of ISO or IEC participate in the development of International Standards through

technical committees established by the respective organization to deal with particular fields of

technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other

international organizations, governmental and non-governmental, in liaison with ISO and IEC, also

take part in the work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC

list of patent declarations received (see patents.iec.ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see www .iso .org/

iso/ foreword .html.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 37, Biometrics.

This second edition cancels and replaces the first edition (ISO/IEC 19795-1:2006), which has been

technically revised.
The main changes compared to the previous edition are as follows:

— Terminology is updated to follow the biometrics vocabulary of ISO/IEC 2382-37:2017;

— Additional detail is provided on testing and reporting of transaction times and computational

workload, and on graphical representation of results.
A list of all parts in the ISO 19795 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/ members .html.
vi © ISO/IEC 2021 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC FDIS 19795-1:2021(E)
Introduction

This document is concerned solely with the scientific technical performance testing of biometric

systems and devices. Technical performance testing seeks to determine error and throughput rates,

with the goal of understanding and predicting the real-world error and throughput performance

of biometric systems. The error rates include both false-positive and false-negative rates, as well as

failure-to-enrol and failure-to-acquire rates across the test population. Throughput rates refer to the

number of individuals processed per unit of time based both on computational speed and human–

machine interaction. These measures are generally applicable to all biometric systems and devices.

Technical performance tests that are modality-specific, for example, fingerprint scanner image quality,

are not considered in this document.

The purpose of this document is to present the requirements and best scientific practices for conducting

and reporting technical performance testing. It is acknowledged that technical performance testing is

only one form of biometric testing. Other types of testing not considered in this document include:

— reliability, availability and maintainability;
— security, including vulnerability;
— conformance;
— safety;
— human factors, including user acceptance;
— cost/benefit;
— privacy regulation conformance.

Biometric technical performance testing can be of three types: technology, scenario and operational

evaluation. Each type of test requires a different protocol and produces different types of results. Other

parts of the ISO/IEC 19795 series provide specific advice and requirements for the development and

use of such different test protocols. This document addresses specific philosophies and principles that

can be applied over a broad range of test conditions.
© ISO/IEC 2021 – All rights reserved vii
---------------------- Page: 7 ----------------------
FINAL DRAFT INTERNATIONAL STANDARD ISO/IEC FDIS 19795-1:2021(E)
Information technology — Biometric performance testing
and reporting —
Part 1:
Principles and framework
1 Scope
This document:

— establishes general principles for testing the performance of biometrics systems in terms of error

rates and throughput rates for purposes including measurement of performance, prediction of

performance, comparison of performance, and verifying conformance with specified performance

requirements;
— specifies performance metrics for biometric systems;

— specifies requirements on the recording of test data and reporting of test results; and

— specifies requirements on test protocols in order to:
— reduce bias due to inappropriate data collection or analytic procedures;
— help achieve the best estimate of field performance for the expended effort;
— improve understanding of the limits of applicability of the test results.

This document is applicable to empirical performance testing of biometric systems and algorithms

through analysis of the comparison scores and decisions output by the system, without requiring

detailed knowledge of the system’s algorithms or of the underlying distribution of biometric

characteristics in the population of interest.

Not within the scope of this document is the measurement of error and throughput rates for people

deliberately trying to subvert the intended operation of the biometric system (e.g. by presentation

attacks).
2 Norm
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.