ISO/IEC TR 29189:2015

Information technology — Biometrics — Evaluation of examiner assisted biometric applications

The purpose of ISO/IEC TR 29189:2015 is to identify and characterize those aspects of performance testing that are unique to examiner assisted biometric applications.

Technologies de l'information — Biométrie — Évaluation des applications biométriques assistées par un examinateur

General Information

• Status: Published
• Publication Date: 08-Jul-2015
• ICS: 35.040 - Information coding
• ICS: 35.240.15 - Identification cards. Chip cards. Biometrics
• Technical Committee: ISO/IEC JTC 1/SC 37 - Biometrics
• Drafting Committee: ISO/IEC JTC 1/SC 37/WG 5 - Biometric testing and reporting
• Current Stage: 9093 - International Standard confirmed
• Completion Date: 10-Feb-2022

Standards Content (Sample)

TECHNICAL ISO/IEC TR
REPORT 29189
First edition
2015-06-15
Information technology — Biometrics
— Evaluation of examiner assisted
biometric applications
Technologies de l’information — Biométrie — Évaluation des
applications biométriques assistées par un examinateur
Reference number
ISO/IEC TR 29189:2015(E)
©
ISO/IEC 2015

---------------------- Page: 1 ----------------------
ISO/IEC TR 29189:2015(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2015, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. [phone]
Fax [phone]
[e-mail]
www.iso.org
ii © ISO/IEC 2015 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC TR 29189:2015(E)

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Terms and definitions . 1
3 Symbols and abbreviated terms . 2
4 Example of an examiner assisted search process . 2
5 Factors to consider when evaluating examiner assisted biometric applications .4
5.1 General . 4
5.2 System-related factors to consider when evaluating examiner assisted
biometric applications . 5
5.2.1 Dependencies in the flow process — Where does the examiner interact
with the system? . 5
5.2.2 System and stage-level performance measurement . 5
5.2.3 Measuring ‘true’ operational performance . 7
5.2.4 The impact of prior probabilities on human performance . 8
5.2.5 Confidence Levels . 8
5.2.6 The impact of automated systems on human performance . 9
5.3 Examiner-related factors to consider when evaluating examiner assisted
biometric applications . 9
5.3.1 An Examiner’s perception of the system’s accuracy . 9
5.3.2 Usability and examiner acceptance .10
5.3.3 Training and expertise .10
5.3.4 Workload .11
5.3.5 Bias in decision making .11
5.3.6 Individual differences between examiners .12
6 Performance evaluation of examiner assisted systems .12
6.1 Types of Evaluation .12
6.2 Performance measures for examiner assisted biometric systems .13
6.2.1 Introduction .13
6.2.2 Measures of accuracy .13
6.2.3 Examiner-assisted performance considerations in watch list scenarios .14
6.2.4 Discrimination and bias .14
6.2.5 Examiner Decision Confidence .14
6.2.6 Processing speed .14
6.3 Usability assessment .15
6.3.1 Introduction .15
6.3.2 Qualitative observations .15
6.3.3 Questionnaires .15
6.3.4 Interviews and focus groups .15
6.4 Reporting results .15
6.5 Applying controls in evaluations .16
6.5.1 Introduction .16
6.5.2 Controls for examiner expertise .16
6.5.3 Controls for examiner decision bias . .16
6.5.4 Controls for the test environment .17
6.5.5 Controls for variations in examiner input .17
6.6 Evaluation challenges .18
6.6.1 Introduction .18
6.6.2 Challenges with testing on a live operational system .18
6.6.3 Challenges in repeatable operational test .18
Bibliography .19
© ISO/IEC 2015 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC TR 29189:2015(E)

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
Details of any patent rights identified during the development of the document will be in the Introduction
and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical Barriers
to Trade (TBT), see the following URL: Foreword — Supplementary information.
The committee responsible for this document is ISO/IEC JTC 1, Information technology, Subcommittee
SC 37, Biometrics.
iv © ISO/IEC 2015 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC TR 29189:2015(E)

Introduction
Biometric identification systems such as those used in forensic applications are typically examiner
assisted and not automated to the extent that most biometric systems are. This is particularly the case
for applications such as latent fingerprint searching where sample quality can be so poor that the system
requires human input. Key processes such as sample capture and preparation, enrolment, template
generation, matching result adjudication, and final decision that would otherwise require minimal
manual intervention are instead heavily reliant on input from experts (fingerprint examiners in the
case of AFIS). These experts can interact with the system at each of these stages to prepare, launch,
and/or review the results of biometric searches. The execution and performance of the “end-to-end”
search process is thus, a combination of the examiner’s role (and capability) and the functionality of the
automated biometric system.
This partially automated approach to biometrics using “examiner assisted” biometric systems provides
value both in assisting the human examiner to perform their role more effectively, and in allowing the
expertise of the human examiner to be exploited to assist the automated matching process. Therefore,
such systems are most likely to be beneficial in non-real time scenarios where the search response is not
necessarily required immediately but the throughput of the system is still high.
Understanding the role of the examiner is crucial, as it impacts on the design of the system, the manner
in which it is used, how it is tested, and how the system performance and its individual subcomponents
are defined and measured.
The main objectives of this Technical Report are to describe the characteristics of examiner assisted
biometric applications and, where appropriate, to contrast such applications with mainstream biometric
applications.
This Technical Report addresses the issues with assessing the system as a whole, or by testing the
examiner assisted and automated elements separately.
© ISO/IEC 2015 – All rights reserved v

---------------------- Page: 5 ----------------------
TECHNICAL REPORT ISO/IEC TR 29189:2015(E)
Information technology — Biometrics — Evaluation of
examiner assisted biometric applications
1 Scope
The purpose of this Technical Report is to identify and characterize those aspects of performance testing
that are unique to examiner assisted biometric applications.
An examiner assisted biometric system has the following characteristics:
— reliant on the interaction and skill of a human examiner for one or more stages of the complete
biometric process, be it data capture, enrolment, template generation, or final decision;
— can incorporate identification functionality, verification functionality, or both;
— will use a combination of the examiner’s input and the functionality of the biometric algorithm to
execute the complete biometric process;
— will likely have inbuilt examination toolsets to assist the human examiner when enrolling biometric
samples or when comparing the match results provided by the biometric algorithm.
Although there is a wide variation in the use of the term “examiner” in the context of an “examiner
assisted biometric system”, as defined in this Technical Report, an “examiner” typically has the following
characteristics:
— field expert in the biometric modality being exploited;
— trained to use the system to an advanced degree of proficiency;
— authorized to override the biometric system’s decisions in particular when accepting or rejecting a
match decision based on their own examination of the biometric samples and the results returned.
Assessing an examiner’s level of expertise is excluded from the scope of this Technical Report. However,
the skill of the examiner does have a major bearing on system performance and vice versa. Measuring
or assessing the ability of an examiner to employ their skills might be necessary to properly evaluate the
performance of an examiner-assisted system.
Other individuals, such as administrative users, or subjects whose biometrics are used within the
system are not considered in this Technical Report. It is outside the scope of this Technical Report to
consider non-expert examiners.
2 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
2.1
examiner
person responsible for examining biometric data and biometric system outputs for the purpose of either
preparing data suitable for a system or confirming, overriding, or modifying a decision output from the
biometric system
Note 1 to entry: This decision output could be a match decision or simply the location of a biometric feature point
(e.g. a fingerprint core and delta points, or the location of eye co-ordinates on a facial image).
© ISO/IEC 2015 – All rights reserved 1

---------------------- Page: 6 ----------------------
ISO/IEC TR 29189:2015(E)

2.2
examiner assisted
feature or quality of a process, application, system, or any other element that refers to the fact that an
examiner takes part by contributing his/her knowledge and expertise
2.3
suspected match
decision state indicating qualified support on the part of an examiner that a match exists, based on the
outcome of the examination process and on the limitations of the relevant comparable data
2.4
suspected non match
decision state indicating qualified support on the part of an examiner that no match exists, based on the
outcome of the examination process and on the limitations of the relevant comparable data
3 Symbols and abbreviated terms
AFIS Automated Fingerprint Identification System
4 Example of an examiner assisted search process
Consider the diagram below in Figure 1 which illustrates at a very high level, some of the basic stages
of a biometric search process. With the exception of the “search” which is fully automated, all other
processes are potentially assisted by the interaction of a human examiner. Figure 2 shows each of the
examiner assisted points in a diagram representing a generic biometric application.
Capture Encode Search Search Response(s) Examiner‘s
Parameters Decision
2
1 3 4 5
Edit
Figure 1 — Basic stages of a biometric search process
2 © ISO/IEC 2015 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC TR 29189:2015(E)

4
3
2
1
5
Figure 2 — A generic biometric application highlighted to indicate the examiner assisted points
To illustrate the importance of the examiner assisted stages, consider the role of forensic AFIS examiners.
These are fingerprint experts, trained specifically to interact with the system, to fully exploit the
functionality of the AFIS in order to prepare, launch, and review the results of the biometric searches.
Their interaction at each of these examiner assisted stages shown in Figure 1 can be described as follows.
a) Capture: An image is scanned by an examiner or imported directly into the system. If multiple
images are available the examiner may select the image(s) that they consider as (the most) suitable
quality for searching.
b) Edit and Encode: The image is displayed for viewing on a monitor and may be enhanced or edited by
the examiner to improve the visibility, and subsequent placement, of features by an examiner. User
interface tools are provided to enable the examiner to manually encode features such as fingerprint
minutiae, cores, deltas, etc. The examiner may also override system decisions about the placement of
features such as minutiae, based on their skill and expertise. Some systems may iterate this process
to gradually improve the quality of the data with each cycle of manual and automated processing.
c) Search Parameters: The examiner may specify search parameters to provide additional data
to the matcher in order to maximise likelihood of the search resulting in a match if one exists in
the database. Finger position, palm region, orientation, or pattern type may typically be input by
the examiner following careful study of the biometric data being searched, based on their domain
specific knowledge.
d) Responses: The matcher threshold may be manually configurable in order to adjust the number of
responses returned. Alternatively, the desired number of responses may be configurable directly,
within some system-defined bounds. Some searches may be assigned a certain level of priority
(over other searches) depending on the importance of the search outcome. An example of this may
be a search conducted on a police system relating to a serious crime.
e) Decision: When the output of the search is returned (typically as a ranked list of potential matches
when used in a forensic context) the examiner compares the enquiry and respondent images in order to
accept or reject possible matches. Even at this stage certain tools on the user interface may be utilised
to assist the examiner in performing this comparison. In some cases the examiner may not be able to
© ISO/IEC 2015 – All rights reserved 3

---------------------- Page: 8 ----------------------
ISO/IEC TR 29189:2015(E)

make an acceptance or rejection decision and may either deem the result indeterminate or return to
the edit and encoding step to initiate an augmented search. Such practice should be documented.
At stages 1 through 3, the aim is to provide additional information to the system that cannot be derived
automatically. In the case of forensic AFIS the fingerprint data submitted for searching is generally of
poor quality, highly varied and thus requires the input of the examiner in order to be able to accurately
search the database. Therefore, the value of an examiner interacting with the system is the direct impact
that their actions have on performance, especially where data quality is severely compromised.
Although forensic AFIS has been chosen as an example, a wide variety of biometric systems or applications
could involve examiner interaction.
The following list provides some examples:
— AFIS — fingerprint matching system, typically using full ten-print enrolments and usually full
10-print probes, often very large scale. Human role is usually to perform a final match/non match
decision from a candidate list
— Forensic AFIS — semi-automated fingerprint matching application, often using 10-print enrolments.
The human role is to mark-up the latent print and make a final match/non match decision.
— Facial recognition — alias or duplicate enrolment detection. The human role is to perform a final
match/non match decision from a candidate list (applications such as visa programs, drivers licences)
— Physical Access Control — a security guard making a human decision of facial match/non
match — for example using printed face on ID card/passport — as part of a secondary check or a
back-up process in the event of the biometric comparison resulting in a reject decision.
— Adjudication processes — Any decision output from a biometric system that is one of ‘Match’, ‘Non
match’, ’Uncertain’ or ‘Suspected Match’, and where all “uncertain” and “suspected” instances are
brought to the attention of a trained human agent to resolve, or will be left in the system in the
suspected state in anticipation that new biometric or other data, or advances in technologies or
changes in policy will allow a resolution of the match.
— Enrolment Quality Checks — a decision, made by a human (possibly aided by automated tools)
following a check to determine if the quality of an enrolment sample(s) is of sufficient quality to
accept, or if the subject needs to retry enrolment.
Forensic AFIS applications (used for latent searching) are reliant on the interaction of a fingerprint
expert at each stage of the complete biometric process. This serves as a good example of an examiner
assisted system as it is well defined or understood in comparison to other examiner assisted applications.
Therefore, it will be used to illustrate many of the points made in this report.
NOTE To demonstrate the contrast of a forensic AFIS system from that of a standard AFIS consider the
scenario of a (civilian) fingerprint system being used for checking identity documents at a point of exit or entry.
The operator of the system might be involved at a number of stages in the overall system functionality — for
example, to assist subjects during the enrolment, or to manually oversee subjects pass through an entry/exit
point controlled by the system. However, the operator in this instance would not be an expert, or be required to
examine the biometric data at the time; rather their actions would be prompted by the output of the system. It
is outside the scope of this technical report to consider such (non-expert) operators or indeed all other users or
administrators.
5 Factors to consider when evaluating examiner assisted biometric applications
5.1 General
Any sound evaluation should begin with a thorough examination of the context in which the biometric
system is operating, as well as the business processes underlying its use. Such an assessment is generally
qualitative in nature, and may consist of interviews or process mapping tools aimed at gaining a sound
understanding of current processes and procedures.
4 © ISO/IEC 2015 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/IEC TR 29189:2015(E)

There are a number of factors to consider when evaluating examiner assisted biometric applications,
and their relative importance varies with application. At each stage where an examiner is involved with
the system, the test design must consider whether this interaction should be specifically addressed or
accounted for in the test. It is beyond the scope of this technical report to make specific recommendations;
however, this technical report will highlight some points to consider when evaluating such systems.
Broadly speaking these considerations can be categorised as ‘system-related’ or ‘examiner-related’ factors.
5.2 System-related factors to consider when evaluating examiner assisted biometric
applications
5.2.1 Dependencies in the flow process — Where does the examiner interact with the system?
Examiner interaction with the system, at any stage, may have an effect on overall processes and
performance. Decisions made at one stage may also have implications for the level of interaction required
by the examiner at subsequent stages. Ultimately, there is a trade-off between increasing reliance on an
examiner, either in part or across the whole end to end process, against the benefits to performance overall.
The Edit and Encode process, described earlier in Clause 4, is an example of this. An examiner may be
required to spend more time editing and encoding an image to pre-process the search for the matcher
to perform better; thereby reducing the time required by the examiner to visually examine images at
the decision stage. Alternatively, a system may be designed to minimise the amount of time allocated for
pre-processing the search, with greater reliance placed on visual examination of search results.
Therefore, evaluations that measure the performance of examiner assisted biometric applications
should take these interdependencies within the overall process into account in order to understand is
there is merit in changing the level of reliance on an examiner at any stage of the process.
The test design should attempt to identify and quantify the level of impact that the examiner’s actions
have on the performance of the system. However, simply removing the examiner from the process may
not be feasible or desirable, and where appropriate it may be better to impose controls around what the
examiner can do at particular stages in order to isolate and understand the impact of their actions on
subsequent stages.
For example in 5.2.2 below the concept of system and stage-level performance is introduced to decompose
the overall process into stages that are automated (partially or fully) and those that are entirely reliant
on the examiner.
5.2.2 System and stage-level performance measurement
5.2.2.1 Introduction
In order to clearly understand the contribution/impact of human input/interaction in the overall
biometric process, it is necessary to decompose the overall process into individually measurable stages.
Then when taken collectively, the overall system-level performance can also be computed. The differences
between 1:1 verification and 1:N identification systems dictate that different performance measures
be described for these two categories. Furthermore, there is a need to describe specific performance
measures for the different automated and examiner assisted stages. The decision matrices described in
the following sections provide examples of determining system and stage-level performance to assess
the stage where the final match decision is made, or overridden, by a human examiner.
In the following tables, green cells indicate correct decisions, and red cells indicate incorrect decisions.
Identification application with human examiner input at the final decision stage only The following describes
the means of defining and computing performance at each stage of an identification application where the
final match/non match decision is made by a human reviewing a candidate list of potential matches.
There are 5 possible outcomes for the automated stage where a candidate list is generated and for which
a true mate biometric reference sample is either previously enrolled (mated) or not enrolled (non-mated)
on the system. See the Table 1 below for details.
© ISO/IEC 2015 – All rights reserved 5

---------------------- Page: 10 ----------------------
ISO/IEC TR 29189:2015(E)

Table 1 — Decision matrix for identification application (Automated stages)
MATEDNON MATED
List generated, Matching candidate ON LIST True (Match) Positive Identiication (n/a)
(known as “Reliability”)
List generated, Matc
...