ISO/IEC DIS 19795-1

Information technology -- Biometric performance testing and reporting

Technologies de l'information -- Essais et rapports de performance biométriques

Standards Content (sample)

DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 19795-1
ISO/IEC JTC 1/SC 37 Secretariat: ANSI
Voting begins on: Voting terminates on:
2019-11-27 2020-02-19
Information technology — Biometric performance testing
and reporting —
Part 1:
Principles and framework
Technologies de l'information — Essais et rapports de performance biométriques —
Partie 1: Principes et canevas
ICS: 35.240.15
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
THEREFORE SUBJECT TO CHANGE AND MAY
NOT BE REFERRED TO AS AN INTERNATIONAL
STANDARD UNTIL PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
BEING ACCEPTABLE FOR INDUSTRIAL,
This document is circulated as received from the committee secretariat.
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
WHICH REFERENCE MAY BE MADE IN
Reference number
NATIONAL REGULATIONS.
ISO/IEC DIS 19795-1:2019(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION. ISO/IEC 2019
---------------------- Page: 1 ----------------------
ISO/IEC DIS 19795-1:2019(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2019

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Fax: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2019 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC DIS 19795-1:2019(E)
Contents Page

Foreword ........................................................................................................................................................................ vii

Introduction ................................................................................................................................................................. viii

1 Scope ....................................................................................................................................................................1

2 Normative references ....................................................................................................................................1

3 Terms, definitions and abbreviated terms .............................................................................................1

4 Conformance .....................................................................................................................................................5

5 General biometric system .............................................................................................................................5

5.1 Conceptual representation of general biometric system ..................................................................5

5.2 Conceptual components of a general biometric system ....................................................................6

Data capture subsystem ................................................................................................................................6

Transmission subsystem ..............................................................................................................................6

Signal processing subsystem .......................................................................................................................7

Data storage subsystem.................................................................................................................................7

Comparison subsystem..................................................................................................................................7

Decision subsystem .........................................................................................................................................7

Administration subsystem ...........................................................................................................................8

Interface to external application ...............................................................................................................8

5.3 Functions of general biometric system ....................................................................................................8

Enrolment ...........................................................................................................................................................8

Verification of a positive biometric claim ...............................................................................................9

Identification .................................................................................................................................................. 10

5.4 Enrolment, verification & identification transactions .................................................................... 11

5.5 Performance measures............................................................................................................................... 12

Error rates ....................................................................................................................................................... 12

Throughput rates .......................................................................................................................................... 12

Types of performance evaluation ........................................................................................................... 13

6 Planning the evaluation ............................................................................................................................. 13

6.1 General ............................................................................................................................................................. 13

6.2 Determine information about the system ........................................................................................... 14

6.3 Controlling factors that influence performance ................................................................................ 15

6.4 Test subject selection .................................................................................................................................. 16

6.5 Test size ............................................................................................................................................................ 17

General ............................................................................................................................................................. 17

Collecting multiple recognition transactions per test subject per system .............................. 18

Requirements on test size ......................................................................................................................... 18

6.6 Multiple tests .................................................................................................................................................. 19

7 Data collection ............................................................................................................................................... 19

7.1 Avoidance of data collection errors ....................................................................................................... 19

7.2 Data and details collected .......................................................................................................................... 20

7.3 Enrolments ...................................................................................................................................................... 21

Enrolment transactions .............................................................................................................................. 21

Enrolment conditions.................................................................................................................................. 21

Enrolment failures and presentation errors ...................................................................................... 22

© ISO/IEC 2019 – All rights reserved
iii
---------------------- Page: 3 ----------------------
ISO/IEC DIS 19795-1:2019(E)

7.4 Comparison trials ......................................................................................................................................... 22

General ............................................................................................................................................................. 22

Collection conditions .................................................................................................................................. 23

Frequency of use ........................................................................................................................................... 23

Systems performing optimisation based on enrolled references .............................................. 24

Systems performing reference adaptation ......................................................................................... 24

Processes for data entry errors and system misuse ........................................................................ 24

Failures to acquire ....................................................................................................................................... 24

Adding test data to the corpus ................................................................................................................. 24

Online comparison trials ........................................................................................................................... 24

Offline comparison trials ........................................................................................................................... 25

Offline non-mated comparison trials when references are dependent ................................... 26

Offline non-mated comparison trials based on comparison of references ............................. 26

Use of samples from multi-capture comparison transactions ..................................................... 26

7.5 Identification trials ...................................................................................................................................... 27

General ............................................................................................................................................................. 27

Identification testing with non-enrolled test subjects ................................................................... 27

Use of Jack-knife approach for identification testing ..................................................................... 27

8 Analyses ........................................................................................................................................................... 27

8.1 General ............................................................................................................................................................. 27

8.2 Performance of biometric enrolment ................................................................................................... 28

Failure-to-enrol rate ................................................................................................................................... 28

Enrolment transaction duration............................................................................................................. 29

8.3 Performance of biometric acquisition .................................................................................................. 29

Failure-to-acquire rate ............................................................................................................................... 29

Acquisition duration ................................................................................................................................... 30

Other aspects of acquisition performance .......................................................................................... 30

8.4 One-to-one comparison performance .................................................................................................. 30

False non-match rate .................................................................................................................................. 30

False match rate ............................................................................................................................................ 30

8.5 Verification system performance metrics ........................................................................................... 31

General ............................................................................................................................................................. 31

False reject rate ............................................................................................................................................ 32

False accept rate ........................................................................................................................................... 32

Verification transaction duration .......................................................................................................... 33

Generalized false reject rate and generalized false accept rate .................................................. 33

8.6 Identification system performance metrics ....................................................................................... 34

General ............................................................................................................................................................. 34

False-negative identification rate .......................................................................................................... 34

False-positive identification rate ........................................................................................................... 34

Selectivity ........................................................................................................................................................ 35

Closed-set test of identification performance ................................................................................... 35

Estimation of identification error rates from verification results ............................................. 35

Predicting identification error rates in larger populations ......................................................... 36

8.7 Analysis of performance across controlled experimental factors ............................................. 36

Longitudinal analyses ................................................................................................................................. 36

Pairwise analyses ......................................................................................................................................... 36

8.8 Detection error trade-off ........................................................................................................................... 36

8.9 Transaction durations ................................................................................................................................ 37

8.10 Computational workload........................................................................................................................... 37

8.11 Uncertainty of estimates ............................................................................................................................ 39

9 Graphical presentation of results........................................................................................................... 39

© ISO/IEC 2019 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC DIS 19795-1:2019(E)

9.1 Score distributions ....................................................................................................................................... 39

General ............................................................................................................................................................. 39

Boxplots ........................................................................................................................................................... 39

9.2 DET plot ............................................................................................................................................................ 40

9.3 CMC plot ........................................................................................................................................................... 41

9.4 Heat maps ........................................................................................................................................................ 44

10 Record keeping .............................................................................................................................................. 45

11 Reporting performance results ............................................................................................................... 46

11.1 Reporting test details .................................................................................................................................. 46

11.2 Summary statistics ....................................................................................................................................... 47

11.3 Reporting enrolment performance ........................................................................................................ 47

11.4 Reporting acquisition performance....................................................................................................... 47

11.5 Reporting one-to-one comparison performance .............................................................................. 48

11.6 Reporting verification system performance ....................................................................................... 48

11.7 Reporting identification system performance ................................................................................... 48

11.8 Reporting performance across factors ................................................................................................. 49

Annex A (informative) Differences between evaluation types ................................................................. 51

Annex B (Informative) Test size and random uncertainty ......................................................................... 52

B.1 Confidence intervals and test size assuming independent identically distributed

comparisons ................................................................................................................................................... 52

B.1.1 Rule of 3 ............................................................................................................................................................ 52

B.1.2 Rule of 30 ......................................................................................................................................................... 52

B.1.3 Number of comparisons to support a claimed error rate .............................................................. 53

B.2 Variance of performance measures as a function of test size....................................................... 53

B.3 Estimates for variance of performance measures ............................................................................ 54

B.3.1 General ............................................................................................................................................................. 54

B.3.2 Variance of observed false non-match rate ......................................................................................... 54

B.3.3 Variance of observed false match rate .................................................................................................. 56

B.4 Estimating confidence intervals .............................................................................................................. 57

B.4.1 General ............................................................................................................................................................. 57

B.4.2 Bootstrap estimates of the variance and confidence intervals .................................................... 58

B.4.3 Subset sampling ............................................................................................................................................ 58

Annex C (informative) Factors influencing performance............................................................................ 60

C.1 General ............................................................................................................................................................. 60

C.2 List of factors .................................................................................................................................................. 60

C.2.1 Population demographics ......................................................................................................................... 60

C.2.2 Application ...................................................................................................................................................... 61

C.2.3 Capture subject anatomy ........................................................................................................................... 61

C.2.4 Capture subject behaviour ........................................................................................................................ 62

C.2.5 Capture subject appearance ..................................................................................................................... 63

C.2.6 Environmental influences ......................................................................................................................... 63

C.2.7 Sensor and hardware .................................................................................................................................. 64

C.2.8 User interface ................................................................................................................................................. 64

C.3 Examples for reporting ............................................................................................................................... 64

C.3.1 Finger position ............................................................................................................................................... 64

C.3.2 Illumination .................................................................................................................................................... 65

C.3.3 Glasses .............................................................................................................................................................. 65

C.3.4 Dirt on platen ................................................................................................................................................. 65

C.3.5 Weather ............................................................................................................................................................ 65

Annex D (Informative) Pre-selection algorithm performance .................................................................. 66

© ISO/IEC 2019 – All rights reserved
---------------------- Page: 5 ----------------------
ISO/IEC DIS 19795-1:2019(E)

Annex E (informative) Identification performance as a function of database size ........................... 68

Annex F (informative) Algorithms for generating DET and CMC ............................................................. 69

F.1 Algorithm for generating DET data points .......................................................................................... 69

F.2 Algorithm for generating CMC data points ......................................................................................... 70

Annex G (informative) DET properties and interpretation ....................................................................... 71

Bibliography ................................................................................................................................................................. 75

© ISO/IEC 2019 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC DIS 19795-1:2019(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are

members of ISO or IEC participate in the development of International Standards through technical

committees established by the respective organization to deal with particular fields of technical activity.

ISO and IEC technical committees collaborate in fields of mutual interest. Other international

organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the

work. In the field of information technology, ISO and IEC have established a joint technical committee,

ISO/IEC JTC 1.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the

different types of document should be noted. This document was drafted in accordance with the editorial

rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights. Details

of any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www.iso.org/patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the World

Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see

www.iso.org/iso/foreword.html.

This document was prepared by Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 37, Biometrics.

This second edition cancels and replaces the first edition (ISO/IEC 19795-1:2006), which has been

technically revised.
The main changes compared to the previous edition are as follows:

⎯ Terminology is updated to follow the biometrics vocabulary of ISO/IEC 2382-37:2017;

⎯ Additional detail on testing and reporting of transaction times and computational workload, and on

graphical representation of results.
A list of all parts in the ISO 19795 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www.iso.org/members.html.
© ISO/IEC 2019 – All rights reserved
vii
---------------------- Page: 7 ----------------------
ISO/IEC DIS 19795-1:2019(E)
Introduction

This document is concerned solely with the scientific “technical performance testing” of biometric

systems and devices. Technical performance testing seeks to determine error and throughput rates, with

the goal of understanding and predicting the real-world error and throughput performance of biometric

systems. The error rates include both false-positive and false-negative rates, as well as failure-to-enrol

and failure-to-acquire rates across the test population. Throughput rates refer to the number of

individuals processed per unit time based both on computational speed and human–machine interaction.

These measures are generally applicable to all biometric systems and devices. Technical performance

tests that are modality-specific — for example, fingerprint scanner image quality — are not considered

in this part of ISO/IEC 19795.

The purpose of this part of ISO/IEC 19795 is to present the requirements and best scientific practices for

conducting and reporting technical performance testing. It is acknowledged that technical performance

testing is only one form of biometric testing. Other types of testing not considered in this part of ISO/IEC

19795 include:
⎯ reliability, availability and maintainability;
⎯ security, including vulnerability;
⎯ conformance;
⎯ safety;
⎯ human factors, including user acceptance;
⎯ cost/benefit;
⎯ privacy regulation compliance.

Biometric technical performance testing can be of three types: technology, scenario and operational

evaluation. Each type of test requires a different proto
...