ISO/IEC 29197:2015
(Main)Information technology — Evaluation methodology for environmental influence in biometric system performance
Information technology — Evaluation methodology for environmental influence in biometric system performance
ISO/IEC 29197:2015 addresses ? fundamental requirements for planning and execution of environmental performance evaluations for biometric systems based on scenario and operational test methodologies, ? specifications to define, establish, and measure specific conditions to assess, including requirements for equipment, ? requirements for establishing a baseline performance in order to compare the influence of environmental parameters, ? a specification of the biometric evaluation including requirements for test population, test protocols, data to record, and test results, and ? procedures for carrying out the overall evaluation. It does not ? determine which parameters should be analysed for a specific biometric modality (This is currently covered in ISO/IEC/TR 19795‑3.), ? specify requirements to perform a vulnerability analysis modifying environmental factors (This is covered by ISO/IEC 19792.), ? classify biometric systems upon performance against different environmental conditions, or ? specify requirements for determining the functional effects of environmental conditions on hardware components (such as corrosion, electrical interference, breakage, etc.) of biometric systems.
Technologies de l'information — Méthodologie de l'évaluation de l'influence environnementale dans la performance d'un système biométrique
General Information
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 29197
First edition
2015-04-01
Information technology — Evaluation
methodology for environmental
influence in biometric system
performance
Technologies de l’information — Méthodologie de l’évaluation de
l’influence environnementale dans la performance d’un système
biométrique
Reference number
ISO/IEC 29197:2015(E)
©
ISO/IEC 2015
---------------------- Page: 1 ----------------------
ISO/IEC 29197:2015(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2015
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2015 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 29197:2015(E)
Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Conformance . 1
3 Normative references . 1
4 Terms and definitions . 2
5 Overview . 3
5.1 Introduction . 3
5.2 Scenario evaluations . 4
5.3 Operational evaluations . 4
6 Evaluation conditions specification . 5
6.1 Introduction . 5
6.2 Definition of evaluation conditions . 5
6.2.1 Description of the evaluation conditions . 5
6.2.2 Types of environmental parameters . 5
6.3 Selection of the evaluation conditions . 6
6.3.1 General. 6
6.3.2 Reference evaluation environment for scenario evaluations . 6
6.3.3 Target evaluation environments . 6
6.4 Measurement and recording of the environmental conditions . 8
6.4.1 General. 8
6.4.2 Instruments to measure and record environmental conditions . 8
6.4.3 Requirements for measuring and recording environmental conditions . 8
7 Biometric scenario evaluation . 9
7.1 Introduction . 9
7.2 Define evaluation objectives . 9
7.3 Environment . 9
7.3.1 Environment for enrolment . 9
7.3.2 Generation and control of the environmental conditions . 9
7.3.3 Biometric system placement .10
7.4 Test population .10
7.4.1 General.10
7.4.2 Size .10
7.4.3 Selection .11
7.4.4 Training and guidance of test subjects .11
7.4.5 Visits .12
7.5 Acclimatization .12
7.6 Levels of effort and decision policies .12
7.7 Exception handling protocols.12
7.8 Data to record and test results .13
7.8.1 General.13
7.8.2 Requirements for recording data .13
7.8.3 Requirements for calculating results .13
7.8.4 Requirements for reporting results .14
7.9 Execution sequence .15
7.9.1 General.15
7.9.2 Pre-test activities .15
7.9.3 Test activities .15
7.9.4 Post-test activities.16
© ISO/IEC 2015 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 29197:2015(E)
8 Operational evaluation .17
8.1 Introduction .17
8.2 Environment .17
8.3 Test period .17
8.4 Data recording processes .17
8.5 Performance measurements .18
8.6 Impostor transactions .18
8.7 Reporting .18
Annex A (informative) Values for environmental parameters .19
Annex B (informative) Test equipment .21
Bibliography .23
iv © ISO/IEC 2015 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 29197:2015(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
Details of any patent rights identified during the development of the document will be in the Introduction
and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical Barriers
to Trade (TBT), see the following URL: Foreword — Supplementary information.
The committee responsible for this document is ISO/IEC JTC 1, Information technology, SC 37, Biometrics.
© ISO/IEC 2015 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 29197:2015(E)
Introduction
The performance of biometric systems can vary according to environmental conditions
(see e.g. ISO/IEC 19795-1:2006, C.2.6, ISO/IEC TR 19795-3:2007, Table 4). Environmental conditions can
affect subjects’ ease of use and comfort in using the systems, subjects’ biometric characteristics, and
also the devices used for acquisition of biometric samples.
This International Standard provides a generic methodology to analyse the influence of environmental
conditions on biometric system performance.
NOTE Environmental conditions can affect several elements involved in the recognition process. However,
the proposed evaluation methodology does not distinguish which of them is affected. The intention of this
methodology is to quantify the overall influence analysing the biometric system performance.
For this International Standard, environmental conditions has to be understood as all atmospheric
parameters (e.g. temperature, humidity) and other physical and chemical phenomena (e.g. illumination,
noise) that can surround the biometric system and influence in its performance. Certain environmental
conditions such as vibration are not dealt with in this International Standard.
These evaluations consist of carrying out a similar “end-to-end” biometric performance evaluation in one
or more predefined environments. These environments can be real (naturally occurring) or modelled
(artificially controlled).
There are two possible ways to carry out an “end-to-end” biometric performance evaluation: performing
scenario evaluations or operational evaluations. In scenario evaluations, biometric systems are
assessed in modelled environments considering a real-world target application and population. These
evaluations are specified as a special case of scenario test based on ISO/IEC 19795-2. An evaluation
performed in a controlled environment can be reproducible, unlike in an operational environment
which uncontrolled parameters can affect the system. In operational evaluations, biometric systems
are analysed in real environments using a target population. These evaluations are specified as a special
case of operational evaluations based on ISO/IEC 19795-6. Operational testing can lack the precision of
scenario testing (in terms of the levels of the environmental parameters), but the testing will benefit
in terms of being operationally realistic. This International Standard provides testing requirements
specific to environmental testing for both kinds of evaluations.
The methodology addresses how to test several environmental factors; however, an evaluation that
conforms to this International Standard can consider as few as one single environmental parameter.
The test parameter or parameters to assess and control has to be previously defined by participants
involved in the evaluation.
The targets of this kind of evaluations include:
— Analyse how one or a combination of environmental factors can affect the biometric system
performance and quantify this influence.
— Analyse how a biometric system works in a specific controlled environment, compared to the same
system working in a reference evaluation environment.
— Analyse how a biometric system behaves in a real environment compared to the same system
working in a modelled environment simulating the operational environment.
The results of these evaluations can inform suppliers and users so that they can assess which
environmental conditions are likely to affect the performance of a biometric system under conditions
they expect to encounter. Results can also indicate whether the particular biometric modality and method
of implementation is appropriate for the situation under which the system is expected to be used. Also
this methodology can be used for examining environmental parameters under which biometric systems
poorly.
vi © ISO/IEC 2015 – All rights reserved
---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 29197:2015(E)
Information technology — Evaluation methodology for
environmental influence in biometric system performance
1 Scope
This International Standard addresses
— fundamental requirements for planning and execution of environmental performance evaluations
for biometric systems based on scenario and operational test methodologies,
— specifications to define, establish, and measure specific conditions to assess, including requirements
for equipment,
— requirements for establishing a baseline performance in order to compare the influence of
environmental parameters,
— a specification of the biometric evaluation including requirements for test population, test protocols,
data to record, and test results, and
— procedures for carrying out the overall evaluation.
This International Standard does not
— determine which parameters should be analysed for a specific biometric modality (This is currently
covered in ISO/IEC/TR 19795-3.),
— specify requirements to perform a vulnerability analysis modifying environmental factors (This is
covered by ISO/IEC 19792.),
— classify biometric systems upon performance against different environmental conditions, or
— specify requirements for determining the functional effects of environmental conditions on
hardware components (such as corrosion, electrical interference, breakage, etc.) of biometric
systems.
2 Conformance
Environmental tests of biometric systems shall be based on scenario evaluations or operational
evaluations. Scenario evaluations shall conform with mandatory requirements of Clause 7 whereas
operational evaluations shall conform with mandatory requirements of Clause 8.
Further, a scenario evaluation that conforms to this International Standard shall analyse at least two
evaluation conditions: the reference evaluation environment and one target evaluation environment.
Both shall consider at least one environmental parameter to assess such as temperature, humidity,
illumination, etc. These evaluation conditions shall be selected, specified, measured, and recorded in
accordance with Clause 6.
3 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 19795-1:2006, Information technology — Biometric performance testing and reporting — Part 1:
Principles and framework
© ISO/IEC 2015 – All rights reserved 1
---------------------- Page: 7 ----------------------
ISO/IEC 29197:2015(E)
ISO/IEC 19795-2, Information technology — Biometric performance testing and reporting — Part 2: Testing
methodologies for technology and scenario evaluation
ISO/IEC 19795-6, Information technology — Biometric performance testing and reporting — Part 6:
Testing methodologies for operational evaluation
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 19795-1 and the following
apply.
4.1
ambient conditions
real environment
common, prevailing, and uncontrolled atmospheric and weather conditions in a room or place
Note 1 to entry: A test described as “conducted at ambient conditions” was performed at whatever conditions
were prevailing at that time on that day.
4.2
baseline performance
performance of a biometric system in a reference evaluation environment
4.3
controlled conditions
climatic and physical conditions that are constrained or managed or kept within certain bounds for the
testing purpose
4.4
environment generator
specialized test equipment used to establish and maintain the controlled conditions of the test
4.5
environmental conditions
all atmospheric parameters (e.g. temperature, humidity) and other physical and chemical phenomena
(e.g. illumination) that can surround the biometric system and influence in its performance
Note 1 to entry: These can be controlled or ambient.
4.6
evaluation condition
environmental condition under which an evaluation is executed
4.7
evaluation configuration
physical layout of the environment in which the biometric system is going to be tested including the
necessary equipment and test instruments for performing tests
4.8
evaluation environment
environment in which the biometric system is evaluated considering the environmental conditions and
the evaluation configuration
4.9
extreme condition
condition that entails very high or very low values of the environmental parameters and may be hostile
for systems operation or even human life
2 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 29197:2015(E)
4.10
instrument
calibrated equipment used to measure and/or record environmental parameters (such as temperature,
humidity, illumination, and sound)
Note 1 to entry: In some cases, instruments have the integrated capability of both measuring and recording
parameters in one piece of equipment.
4.11
measuring point
specific value for an environmental parameter that is being assessed
4.12
operational environment
conditions under which the biometric system is expected to operate
Note 1 to entry: This is a combination of environmental factors and operational procedures.
4.13
reference evaluation environment
evaluation environment in which the biometric system is analysed to obtain baseline performance
metrics
4.14
set point
specific value for an environmental parameter that is being controlled
4.15
target evaluation environment
evaluation environment in which the biometric system is analysed to obtain performance metrics for
studying the influence of certain environmental conditions
5 Overview
5.1 Introduction
An environmental evaluation is a set of trials carried out to analyse the biometric performance of one
or more biometric systems working in particular environmental conditions to determine the effect of
one or more of the conditions. These may be executed as one of two types of biometric evaluations:
scenario evaluations and operational evaluations. Scenario evaluations analyse “end to end” biometric
system performance in a modelled environment whereas operational evaluations analyse “end to end”
biometric system performance in an uncontrolled operational environment.
Once the biometric evaluation is carried out in each of the different evaluation conditions, a group of
performance metrics will be calculated. These metrics are specific for these evaluation conditions.
The number of evaluation conditions to analyse will depend on the following aspects:
— if the evaluation environment is real or modelled and how many variations of environmental
parameters are feasible;
— if the objective is analysing the influence of one or more environmental parameters separately and
how many measuring points are going to be examined per each parameter;
— if the objective is checking how a biometric system works in a particular environment.
These aspects are not independent. A decision on conducting a scenario or operational evaluation
will depend upon the objectives of the environmental evaluation, the number of parameters to assess,
and whether the environment is real or modelled. The following subclauses describe both kinds of
environmental evaluations and their main characteristics.
© ISO/IEC 2015 – All rights reserved 3
---------------------- Page: 9 ----------------------
ISO/IEC 29197:2015(E)
5.2 Scenario evaluations
Scenario evaluations consist of defining one or more specific evaluation conditions as well as a reference
evaluation environment and carrying out a similar scenario test in each of them. In these evaluations, the
biometric system and environmental parameters will be recorded at the same time in order to determine
error rates and throughput rates for the specific evaluation conditions. This approach provides insights
into the degree to which biometric systems are influenced by one or more environmental parameters. A
schema of this evaluation methodology is shown in Figure 1.
Figure 1 — Scenario Test Methodology illustration
Scenario evaluations are developed in a controlled environment usually in a test facility. These
evaluations are suitable for
— measuring the influence of one or more controllable environmental parameters, or
— analysing how one or more biometric systems perform in a specific environment whose parameters
are set to defined values or kept inside a particular range.
In both cases, the environmental parameters being assessed, their corresponding measuring points or
ranges, and the environmental parameters being controlled shall be determined prior to the test.
5.3 Operational evaluations
Operational evaluations consist of determining a specific environment and selecting one or more
environmental parameters to assess. In these evaluations, the relevant biometric system performance
and the selected environmental parameters to assess will be recorded at the same time. This approach
provides insights into the degree to which biometric system performance is influenced by the assessed
parameters of the operational environment.
Operational evaluations are carried out in an uncontrolled environment. When performing this type of
evaluation, it may be difficult to isolate the impact of environmental parameters on users, algorithms,
or equipment; the impact is measured for the system as a whole.
4 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC 29197:2015(E)
Environmental conditions should reflect those typically encountered when the system is not under
evaluation. Thus, an operational test may permit operationally viable practices such as use of a canopy
for outdoor collection of facial images. Therefore, for this evaluation, the operational environment
including physical layout as well as the environmental parameters that are going to be assessed shall be
described.
In addition, it will be necessary that evaluators specify which evaluation conditions are of interest. It
will require conducting a preliminary analysis of the operational environment for knowing the potential
values of the environmental parameters.
This analysis shall be carried out before planning the operational evaluation in order to determine the
following aspects:
— instruments for measuring and recording the environmental parameters to assess;
— the localization of such instruments;
— the definition of the reference evaluation environment.
6 Evaluation conditions specification
6.1 Introduction
Biometric systems can be tested in different evaluation conditions in order to analyse whether their
performance is influenced by environmental conditions or not. This Clause defines requirements for
defining and measuring such evaluation conditions for all potential environmental parameters that can
be tested during this kind of evaluations.
6.2 Definition of evaluation conditions
6.2.1 Description of the evaluation conditions
In general, the evaluation environmental conditions shall be defined considering the following two
aspects:
— Environmental parameters to assess: These are the environmental cond
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.