ISO/IEC 19795-5:2011
(Main)Information technology — Biometric performance testing and reporting — Part 5: Access control scenario and grading scheme
Information technology — Biometric performance testing and reporting — Part 5: Access control scenario and grading scheme
ISO/IEC 19795-5:2011 specifies a framework for testing and a grading scheme for reporting the performance of a biometric system suitable for use in access control applications. It also allows for specifying application performance requirements in terms of the required performance of the biometric component of the access control system. It specifies the environment in which and the means by which testing will be performed and how the results will be reported. The grading scheme takes a conservative approach, using statistical analysis and confidence intervals to support the claim that the device performance is at least as good as the graded performance indicates. ISO/IEC 19795-5:2011 addresses conventional access control circumstances, and unusual or extreme circumstances are not within its scope.
Technologies de l'information — Essais et rapports de performance biométriques — Partie 5: Plan de classement pour évaluation de scénario de contrôle d'accès
General Information
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 19795-5
First edition
2011-03-01
Information technology — Biometric
performance testing and reporting —
Part 5:
Access control scenario and grading
scheme
Technologies de l'information — Essais et rapports de performance
biométriques —
Partie 5: Plan de classement pour évaluation de scénario de contrôle
d'accès
Reference number
ISO/IEC 19795-5:2011(E)
©
ISO/IEC 2011
---------------------- Page: 1 ----------------------
ISO/IEC 19795-5:2011(E)
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2011
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2011 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 19795-5:2011(E)
Contents Page
Foreword .v
Introduction.vi
1 Scope.1
2 Conformance .2
3 Normative references.2
4 Terms and definitions .2
5 Definition of testing scenario.3
5.1 Overview.3
5.2 Relationship of biometric system / subsystem to access control system.3
5.3 Evaluation metrics overview .5
5.4 Evaluation approach .5
5.4.1 Tests .5
5.4.2 Universality of the test.6
5.4.3 Levels of effort and decision policies .6
5.4.4 Controlled Indoor Environment .6
5.5 Crew characteristics and management.7
5.5.1 Crew demographics .7
5.5.2 Crew size .8
5.5.3 Test crew selection .8
5.5.4 Test crew training.9
5.5.5 Operator - crew member interaction .9
5.5.6 Habituation.9
5.6 Privacy.9
5.6.1 General .9
5.6.2 Crew identity protection .9
5.6.3 Data protection .10
5.6.4 Proprietary information.10
6 Testing approach and conduct .10
6.1 Planning .10
6.1.1 General .10
6.1.2 Test objectives.10
6.1.3 Inputs to and outputs from the test process.10
6.1.4 Concept of operations .10
6.1.5 Adherence to native system operations .11
6.2 General test approach.11
6.2.1 General .11
6.2.2 Pre-test activities.12
6.2.3 System operability verification .14
6.2.4 Data collection.14
6.2.5 Problem reporting and tracking.15
6.2.6 Post-test briefing.16
6.3 Testing methodology .16
6.3.1 Introduction.16
6.3.2 Enrolment transactions and results generation .17
6.3.3 Verification attempts, transactions, and results generation.17
6.3.4 Enrolment and verification temporal separation .18
6.3.5 Impostor tests.20
6.4 Errors and exception cases .20
6.5 Incremental performance evaluations.21
© ISO/IEC 2011 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 19795-5:2011(E)
7 Grading and reporting. 21
7.1 Grading . 21
7.1.1 Data analysis. 21
7.1.2 Using statistical analysis methods. 21
7.1.3 Performance measures . 21
7.1.4 Grading of matching performance illustration . 25
7.1.5 Uses (of grading) . 25
7.2 Documentation requirements and control . 26
7.2.1 General. 26
7.2.2 Test control . 26
7.3 Reporting performance results . 27
7.3.1 Reporting requirements . 27
7.3.2 Report structure. 28
Annex A (informative) Grading information . 29
A.1 Equivalence of tests . 29
A.2 Comparison of test results . 29
A.3 Grading values for enrolment performance. 29
A.4 Grading values for matching performance . 30
A.5 Grading illustration shown in Figure A.1 . 30
A.6 Grading values for transaction time performance . 31
A.7 Defining system requirements as in Table 7 . 31
Annex B (normative) Statistical methods for estimation of confidence bounds graded test metrics. 33
B.1 Correlated binary method. 33
B.2 Beta distribution method . 34
B.3 Z-statistic. 35
Bibliography. 36
iv © ISO/IEC 2011 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 19795-5:2011(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 19795-5 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 37, Biometrics.
ISO/IEC 19795 consists of the following parts, under the general title Information technology — Biometric
performance testing and reporting:
⎯ Part 1: Principles and framework
⎯ Part 2: Testing methodologies for technology and scenario evaluation
⎯ Part 3: Modality-specific testing [Technical report]
⎯ Part 4: Interoperability performance testing
⎯ Part 5: Access control scenario and grading scheme
⎯ Part 6: Testing methodologies for operational evaluation
⎯ Part 7: Testing of on-card biometric comparison algorithms
© ISO/IEC 2011 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 19795-5:2011(E)
Introduction
This part of ISO/IEC 19795 is concerned solely with the scientific “technical performance testing” of biometric
systems and subsystems to be used for access control. Technical performance testing seeks to determine
error rates and transaction times with the goal of understanding and predicting the real-world error and
transaction times of a biometric system. The error rates include false accept rate, and false reject rate, as well
as failure to enrol (FTE) and failure to acquire (FTA) rates across the test population. These measures are
generally applicable to all access control systems that contain a biometric verification subsystem.
This part of ISO/IEC 19795 defines a testing framework with the following fundamental aspects.
⎯ This part of ISO/IEC 19795 was conceived to be a framework for a general- or multi-purpose test: “one
size fits many (but not all)”. The focus is limited to access control applications.
⎯ The framework is suitable as both a requirements statement and an evaluation report.
⎯ The general-purpose nature of this part of ISO/IEC 19795 is centred on the common access control
application requirements, and acknowledges the fact that this framework will not be suitable for
specialized applications (very high levels of protection, specialized user populations like the elderly,
students, etc.). Specialized applications will warrant specialized testing processes.
⎯ The perceived benefit of the general- or multi-purpose test is economy. The supplier can submit to one
testing process, and many potential customers can utilize the results, interpreting the suitability of the
device (based on the results) for their application.
This testing framework assigns grades representing the tested level of performance, and these grades include
a statistical confidence taking the conservative approach, that is, the performance of the system is at least as
good as the grade indicated (at the 90% confidence level). Using the grading scheme to specify a required
performance level of a system needs to take into account this conservative approach.
It is acknowledged that technical performance testing is only one form of biometric testing. Other types of
testing not considered in this part of ISO/IEC 19795 include the following:
⎯ reliability, availability and maintainability;
⎯ security, including vulnerability;
⎯ human factors, including user acceptance;
⎯ environmental;
⎯ safety;
⎯ cost/benefit;
⎯ privacy regulation compliance.
Methods and philosophies for these other types of tests are currently being considered internationally by a
broad range of groups.
The purpose of this part of ISO/IEC 19795 is to capture the current understanding by the biometrics
community of requirements and best scientific practices for conducting performance testing towards the end of
providing consistent, structured evaluations of biometric systems intended for use in access control
applications. The framework defined in this part of ISO/IEC 19795 has utility as a method for defining user
requirements, for specifying the extent of performance evaluation, for conducting and for reporting.
vi © ISO/IEC 2011 – All rights reserved
---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 19795-5:2011(E)
Information technology — Biometric performance testing and
reporting —
Part 5:
Access control scenario and grading scheme
1 Scope
This part of ISO/IEC 19795:
⎯ defines a common biometric access control scenario for use in scenario evaluation of biometric
verification systems;
⎯ provides a grading scheme for expressing quantitative biometric system requirements and performance
levels;
⎯ provides a common basis for conducting scenario evaluations to demonstrate that specified performance
grades are being achieved which is adaptable to particular testing facilities and to specific biometric
systems.
This part of ISO/IEC 19795 is applicable to performance testing of biometric systems without detailed
knowledge of the comparison algorithms or of the underlying distribution of biometric characteristics in the
population of interest.
The minimum false accept rate (FAR) tested by this part of ISO/IEC 19795 is 0.1%. If a lower FAR is required,
customized testing (outside the scope of this part of ISO/IEC 19795) might be appropriate, and needs to be
fully compliant with ISO/IEC 19795-2.
This part of ISO/IEC 19795 addresses testing a biometric system for physical access control, and the
suitability of the testing for logical access devices needs to be determined on a case-by-case basis.
Not within the scope of this part of ISO/IEC 19795 is the measurement of error and throughput rates for
people deliberately trying to circumvent correct recognition by the biometric system (i.e. active impostors). In
addition, this part of ISO/IEC 19795 does not assess the following:
⎯ reliability, availability and maintainability;
⎯ security, including vulnerability;
⎯ human factors, including user acceptance;
⎯ environmental impacts;
⎯ safety;
⎯ cost/benefit/suitability;
⎯ privacy regulation compliance.
These assessments are the responsibility of the procuring authority.
© ISO/IEC 2011 – All rights reserved 1
---------------------- Page: 7 ----------------------
ISO/IEC 19795-5:2011(E)
2 Conformance
A test conforms to this part of ISO/IEC 19795 if the scenario used (including test crew demographics,
environmental controls, time separation between enrolment and revisit, numbers of attempts and transactions),
test conduct, and test reporting all conform to the mandatory requirements in Clauses 5 through 7.
3 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced
document (including any amendments) applies.
ISO/IEC 19795-1:2006, Information technology — Biometric performance testing and reporting — Part 1:
Principles and framework
ISO/IEC 19795-2:2007, Information technology — Biometric performance testing and reporting — Part 2:
Testing methodologies for technology and scenario evaluation
ISO/IEC TR 19795-3, Information technology — Biometric performance testing and reporting — Part 3:
Modality-specific testing
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 19795-1 and the following apply.
4.1
access control system
ACS
entire electro-mechanical suite that performs the granting or denying of access at controlled entry points of a
facility
4.2
biometric subsystem
portion of a biometric system that is present at each access entry point, including the biometric sensor or
sampling subsystem
4.3
grade levels
measurement associated with the quantified levels of biometric subsystem performance
NOTE Grade levels are defined, ranging from 0 to 3, or 0 to 6. It is possible that additional grade levels above these
values will be defined at a future date.
4.4
FAR level
scale for the relative level of resistance to false accepts in a form associated with three specific false accept
rate (FAR) values
4.5
transaction time
time required for the biometric system portion of an access control transaction
NOTE Transaction time is measured in seconds.
2 © ISO/IEC 2011 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 19795-5:2011(E)
5 Definition of testing scenario
5.1 Overview
The goal of testing and evaluating biometric access control systems against the standard set of criteria
documented in this part of ISO/IEC 19795 is to ensure that the technical performance of every biometric
access control system is evaluated fairly, accurately and equivalently.
Testing shall be performed in a consistent, unbiased manner under conditions that are well understood and
documented. Test controls shall be applied to ensure reproducible test results to the most practical extent
possible (considering the involvement of human crew members). To accomplish this, every candidate
biometric access control system shall be tested in accordance with the same general test protocol.
The procedures to be used shall be based upon a “framework” consisting of specific metrics extracted from
biometric system operations and accompanying evaluation criteria which provides for graded evaluation
against different levels of false accept rate. The evaluation framework shall accommodate biometric
subsystems that output similarity scores or that output only the final match/no match decision.
NOTE 1 Throughout this part of ISO/IEC 19795, where reference is made to similarity scores, it should be understood
that for those test results in the form of decision output, the equivalent, suitable process is applied.
NOTE 2 Throughout this part of ISO/IEC 19795, where reference is made to similarity scores, it should be understood
that devices that generate dissimilarity scores will be accommodated by making the appropriate threshold comparison
matching decisions.
To facilitate the testing of a specific biometric access control system, a specialized biometric test procedure
shall be developed. It shall be identical to the general procedure with the exception that any additional
information (for example sliding the cover to allow placement of a finger to a sensor) needed in the real-world
operation of a particular biometric access control system shall be identified.
5.2 Relationship of biometric system / subsystem to access control system
A biometric access control system is an access control system that contains a biometric system as a
subsystem. This biometric system can be, for instance:
⎯ a verification or identification system with centralized biometric template storage;
⎯ a verification system with decentralized biometric template storage in the biometric subsystem; or
⎯ a verification system with localized biometric template storage (e.g. on an ID card).
NOTE 1 The evaluation of identification performance metrics is outside the scope of this part of ISO/IEC 19795.
Figure 1 illustrates the components and information flows in a generic access control system that includes a
biometric system. Following Figure 1 is a key to the circled letters representing information flows. Real
deployed systems may vary from this general model.
© ISO/IEC 2011 – All rights reserved 3
---------------------- Page: 9 ----------------------
ISO/IEC 19795-5:2011(E)
Figure 1 — Generic Biometric Access Control System
A description of the information flow in Figure 1 is as follows:
A. Biographical information: applicant-supplied information (name, address, etc.) obtained during Access
Control System (ACS) enrolment via the ACS Processor. This flow is part of a typical legacy ACS.
B. Biometric characteristic (trait): the body part or human behaviour presented by the applicant to the
biometric sensor during enrolment (e.g. fingerprint, iris, voice, signature). This flow may also include any
interactions between applicant and sensor such as indicator lights or audio feedback.
NOTE 2 An applicant becomes a user only after the enrolment process is completed and access privileges are granted
by the access control authority.
C. Token (ID card): any form of machine-readable credential presented by the user to the ID reader to claim
an identity.
D. Biometric trait: the body part or human behaviour presented by the user to the biometric sensor during an
access transaction (e.g., fingerprint, iris, voice, or signature). This flow may also include any interactions
between user and sensor such as indicator lights or audio feedback.
E. User identity code: (ID number, card number, ACS ID) read from the token by the ID reader and sent to
the biometric processor as the claim of identity. This flow also includes user template data for template on
card architectures.
F. Biometric template data: from enrolment database to biometric processor (for implementations using
server-stored templates). This flow is architecture-specific, may be per user transaction or periodic pre-loads.
4 © ISO/IEC 2011 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC 19795-5:2011(E)
G. Biometric decision: Yes/No indication (electrical signal or message) from biometric processor to panel
conveying the result of the user verification transaction.
H. User identity code: (ID number, card number, ACS ID) read from the token by the ID reader and sent to
the panel for the ACS to determine access privilege. This flow is part of a typical legacy ACS.
I. Lock control: electrical signal from the panel used to command the door electro-mechanical locking
mechanisms. This flow may also include other signals such as door-open indicators, emergency lock override,
etc. This flow is part of a typical legacy ACS.
J. ACS network data: (physical) communication channel (Ethernet, RS485, etc.) enabling data interchange
between the panel, ACS processor, and ACS database. The ACS network (logically) depends upon site-
specific implementation, and includes a user identity code from panel and user access authorization from ACS
processor.
5.3 Evaluation metrics overview
The framework is based on the necessary and sufficient metrics for evaluation of a biometric system for use in
an access control application. These metrics are:
⎯ (Single-attempt) false reject rates (FRR ) at specific values of FAR;
1
⎯ (Transaction-level) false reject rates (FRR) at specific values of FAR;
⎯ failure to enrol rate (FTE);
⎯ verification transaction time.
To serve many (not all) applications, a range of protection levels, expressed as specific values of FAR, shall
be used in this framework.
For each metric, the framework establishes a quantitative grading scheme, using numerical grades, ranging
from 0 to 3 (or 0 to 6 for FRR), where a higher score shall indicate better performance and a lower score shall
indicate poorer performance. In Clause 7, the metrics are fully defined and the quantitative grading values are
established.
NOTE 1 Different metrics may have different grading
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.