ISO/IEC 19795-2:2007
(Main)Information technology — Biometric performance testing and reporting — Part 2: Testing methodologies for technology and scenario evaluation
Information technology — Biometric performance testing and reporting — Part 2: Testing methodologies for technology and scenario evaluation
ISO/IEC 19795-2:2007 addresses two specific biometric performance testing methodologies: technology and scenario evaluation. The large majority of biometric tests are of one of these two generic evaluation types. Technology evaluations evaluate enrolment and comparison algorithms by means of previously collected corpuses, while scenario evaluations evaluate sensors and algorithms by processing of samples collected from Test Subjects in real time. The former is intended for generation of large volumes of comparison scores and candidate lists indicative of the fundamental discriminating power of an algorithm. The latter is intended for measurement of performance in modeled environments, inclusive of Test Subject-system interactions. ISO/IEC 19795-2:2007 provides requirements and recommendations on data collection, analysis and reporting specific to the two primary types of evaluation: technology evaluation and scenario evaluation. It specifies requirements in the following areas: development and full description of protocols for technology and scenario evaluations; execution and reporting of biometric evaluations reflective of the parameters associated with biometric evaluation types.
Technologies de l'information — Essais et rapports de performance biométriques — Partie 2: Méthodologies d'essai pour l'évaluation des technologies et du scénario
General Information
Relations
Buy Standard
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 19795-2
First edition
2007-02-01
Information technology — Biometric
performance testing and reporting —
Part 2:
Testing methodologies for technology
and scenario evaluation
Technologies de l'information — Essais et rapports de performance
biométriques —
Partie 2: Méthodologies d'essai pour l'évaluation des technologies et du
scénario
Reference number
©
ISO/IEC 2007
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO/IEC 2007
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2007 – All rights reserved
Contents Page
Foreword. iv
Introduction . v
1 Scope .1
2 Conformance.1
3 Normative references .1
4 Terms and definitions .2
4.1 Biometric data.2
4.2 Components of a biometric system.2
4.3 User interaction with a biometric system .2
4.4 Performance measures .3
5 Overview of technology evaluations and scenario evaluations.3
6 Technology evaluation.6
6.1 Test design.6
6.2 Assembling an appropriate test corpus.8
6.3 Performance measurement .11
6.4 Reporting .16
7 Scenario evaluation.18
7.1 Test design.18
7.2 Test crew .23
7.3 Performance measurement .24
7.4 Reporting .26
8 Other issues applicable to technology and scenario evaluations .29
8.1 Parties to a test .29
8.2 Fairness .29
8.3 Basis for inclusion of test systems .29
8.4 Use of Frequently Asked Questions.30
8.5 Legal issues .30
8.6 Release of test source code .30
8.7 Supplier comment on test report .30
Annex A (informative) Phases and activities for primary technology test types .31
Annex B (informative) Relationship between presentations, attempts, and transactions .37
Annex C (informative) Reporting effort levels.38
Annex D (informative) Client-server testing .40
Annex E (informative) Comparing results across systems in multi-system tests .41
© ISO/IEC 2007 – All rights reserved iii
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 19795-2 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 37, Biometrics.
ISO/IEC 19795 consists of the following parts, under the general title Information technology — Biometric
performance testing and reporting:
⎯ Part 1: Principles and framework
⎯ Part 2: Testing methodologies for technology and scenario evaluation
The following parts are under preparation:
⎯ Part 3: Modality-specific testing [Technical Report]
⎯ Part 4: Performance and interoperability testing of data interchange formats
⎯ Part 5: Performance of biometric access control systems
iv © ISO/IEC 2007 – All rights reserved
Introduction
This part of ISO/IEC 19795 addresses two specific biometric performance testing methodologies: technology
and scenario evaluation. The large majority of biometric tests are of one of these two generic evaluation types.
Technology evaluations evaluate enrolment and comparison algorithms by means of previously collected
corpuses, while scenario evaluations evaluate sensors and algorithms by processing of samples collected
from Test Subjects in real time. The former is intended for generation of large volumes of comparison scores
and candidate lists indicative of the fundamental discriminating power of an algorithm. The latter is intended
for measurement of performance in modeled environments, inclusive of Test Subject-system interactions.
This part of ISO/IEC 19795 builds on requirements and best practices specified in ISO/IEC 19795-1, which
addresses specific philosophies and principles that can be applied over a broad range of test conditions.
This part of ISO/IEC 19795 is meant to provide biometric system developers, deployers and end users with
mechanisms for design, execution and reporting of biometric performance tests in a fashion that allows
meaningful benchmarking of biometric performance within and across technologies, usage scenarios and
environments.
© ISO/IEC 2007 – All rights reserved v
INTERNATIONAL STANDARD ISO/IEC 19795-2:2007(E)
Information technology — Biometric performance testing and
reporting —
Part 2:
Testing methodologies for technology and scenario evaluation
1 Scope
This part of ISO/IEC 19795 provides requirements and recommendations on data collection, analysis and
reporting specific to two primary types of evaluation: technology evaluation and scenario evaluation.
This part of ISO/IEC 19795 specifies requirements in the following areas:
⎯ development and full description of protocols for technology and scenario evaluations;
⎯ execution and reporting of biometric evaluations reflective of the parameters associated with biometric
evaluation types.
2 Conformance
A test shall claim conformance to either the technology evaluation or scenario evaluation clauses of this part
of ISO/IEC 19795.
The set of clauses to which a scenario test shall conform differs from the set of clauses to which a technology
test shall conform. In addition, the set of clauses to which an identification-system test shall conform differs
from the set of clauses to which a verification-system test shall conform. To conform to this part of
ISO/IEC 19795, an evaluation shall conform to clauses of this part of ISO/IEC 19795 as shown in Table 1.
Table 1 — Conformance for evaluation methodologies and comparison types
Evaluation methodology Comparison type Required clauses
Technology or scenario Identification or verification Clauses 5 and 8
Technology Identification All of Clause 6, except 6.3.3
Technology Verification All of Clause 6, except 6.3.4
Scenario Identification All of Clause 7, except 7.3.4
Scenario Verification All of Clause 7, except 7.3.5
3 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced
document (including any amendments) applies.
ISO/IEC 19795-1, Information technology — Biometric performance testing and reporting — Part 1: Principles
and framework
© ISO/IEC 2007 – All rights reserved 1
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 19795-1:2006 and the following
apply.
4.1 Biometric data
4.1.1
biometric reference
〈template, model〉 user’s stored reference measure based on features extracted from enrolment samples
4.2 Components of a biometric system
4.2.1
feature extractor
apparatus that extracts features from a sample
4.2.2
biometric reference generator
apparatus that transforms a sample into a biometric reference
4.3 User interaction with a biometric system
4.3.1
acclimatization
reduction, over the course of an evaluation, in a temporal condition of a biometric characteristic that may
impact the ability of a sensor to process a sample
4.3.2
effort level
number of presentations, attempts or transactions needed to successfully enrol or match in a biometric system
4.3.3
enrolment attempt
submission of one or more biometric samples for a Test Subject for the purpose of enrolment in a biometric
system
NOTE 1 One or more enrolment attempts may be permitted or required to constitute an enrolment transaction. An
enrolment attempt may comprise one or more enrolment presentations.
NOTE 2 See Annex B for illustration of the relationship between presentation, attempt and transaction.
4.3.4
enrolment attempt limit
maximum number of attempts, or the maximum duration, a Test Subject is permitted before an enrolment
transaction is terminated
4.3.5
enrolment presentation
submission of an instance of a biometric characteristic for a Test Subject for the purpose of enrolment
NOTE One or more enrolment presentations may be permitted or required to constitute an enrolment attempt. An
enrolment presentation may or may not result in an enrolment attempt.
4.3.6
enrolment presentation limit
maximum number of presentations, or the maximum duration, a Test Subject is permitted before an enrolment
attempt is terminated
2 © ISO/IEC 2007 – All rights reserved
...
INTERNATIONAL ISO/IEC
STANDARD 19795-2
First edition
2007-02-01
Information technology — Biometric
performance testing and reporting —
Part 2:
Testing methodologies for technology
and scenario evaluation
Technologies de l'information — Essais et rapports de performance
biométriques —
Partie 2: Méthodologies d'essai pour l'évaluation des technologies et du
scénario
Reference number
©
ISO/IEC 2007
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO/IEC 2007
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2007 – All rights reserved
Contents Page
Foreword. iv
Introduction . v
1 Scope .1
2 Conformance.1
3 Normative references .1
4 Terms and definitions .2
4.1 Biometric data.2
4.2 Components of a biometric system.2
4.3 User interaction with a biometric system .2
4.4 Performance measures .3
5 Overview of technology evaluations and scenario evaluations.3
6 Technology evaluation.6
6.1 Test design.6
6.2 Assembling an appropriate test corpus.8
6.3 Performance measurement .11
6.4 Reporting .16
7 Scenario evaluation.18
7.1 Test design.18
7.2 Test crew .23
7.3 Performance measurement .24
7.4 Reporting .26
8 Other issues applicable to technology and scenario evaluations .29
8.1 Parties to a test .29
8.2 Fairness .29
8.3 Basis for inclusion of test systems .29
8.4 Use of Frequently Asked Questions.30
8.5 Legal issues .30
8.6 Release of test source code .30
8.7 Supplier comment on test report .30
Annex A (informative) Phases and activities for primary technology test types .31
Annex B (informative) Relationship between presentations, attempts, and transactions .37
Annex C (informative) Reporting effort levels.38
Annex D (informative) Client-server testing .40
Annex E (informative) Comparing results across systems in multi-system tests .41
© ISO/IEC 2007 – All rights reserved iii
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 19795-2 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 37, Biometrics.
ISO/IEC 19795 consists of the following parts, under the general title Information technology — Biometric
performance testing and reporting:
⎯ Part 1: Principles and framework
⎯ Part 2: Testing methodologies for technology and scenario evaluation
The following parts are under preparation:
⎯ Part 3: Modality-specific testing [Technical Report]
⎯ Part 4: Performance and interoperability testing of data interchange formats
⎯ Part 5: Performance of biometric access control systems
iv © ISO/IEC 2007 – All rights reserved
Introduction
This part of ISO/IEC 19795 addresses two specific biometric performance testing methodologies: technology
and scenario evaluation. The large majority of biometric tests are of one of these two generic evaluation types.
Technology evaluations evaluate enrolment and comparison algorithms by means of previously collected
corpuses, while scenario evaluations evaluate sensors and algorithms by processing of samples collected
from Test Subjects in real time. The former is intended for generation of large volumes of comparison scores
and candidate lists indicative of the fundamental discriminating power of an algorithm. The latter is intended
for measurement of performance in modeled environments, inclusive of Test Subject-system interactions.
This part of ISO/IEC 19795 builds on requirements and best practices specified in ISO/IEC 19795-1, which
addresses specific philosophies and principles that can be applied over a broad range of test conditions.
This part of ISO/IEC 19795 is meant to provide biometric system developers, deployers and end users with
mechanisms for design, execution and reporting of biometric performance tests in a fashion that allows
meaningful benchmarking of biometric performance within and across technologies, usage scenarios and
environments.
© ISO/IEC 2007 – All rights reserved v
INTERNATIONAL STANDARD ISO/IEC 19795-2:2007(E)
Information technology — Biometric performance testing and
reporting —
Part 2:
Testing methodologies for technology and scenario evaluation
1 Scope
This part of ISO/IEC 19795 provides requirements and recommendations on data collection, analysis and
reporting specific to two primary types of evaluation: technology evaluation and scenario evaluation.
This part of ISO/IEC 19795 specifies requirements in the following areas:
⎯ development and full description of protocols for technology and scenario evaluations;
⎯ execution and reporting of biometric evaluations reflective of the parameters associated with biometric
evaluation types.
2 Conformance
A test shall claim conformance to either the technology evaluation or scenario evaluation clauses of this part
of ISO/IEC 19795.
The set of clauses to which a scenario test shall conform differs from the set of clauses to which a technology
test shall conform. In addition, the set of clauses to which an identification-system test shall conform differs
from the set of clauses to which a verification-system test shall conform. To conform to this part of
ISO/IEC 19795, an evaluation shall conform to clauses of this part of ISO/IEC 19795 as shown in Table 1.
Table 1 — Conformance for evaluation methodologies and comparison types
Evaluation methodology Comparison type Required clauses
Technology or scenario Identification or verification Clauses 5 and 8
Technology Identification All of Clause 6, except 6.3.3
Technology Verification All of Clause 6, except 6.3.4
Scenario Identification All of Clause 7, except 7.3.4
Scenario Verification All of Clause 7, except 7.3.5
3 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced
document (including any amendments) applies.
ISO/IEC 19795-1, Information technology — Biometric performance testing and reporting — Part 1: Principles
and framework
© ISO/IEC 2007 – All rights reserved 1
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 19795-1:2006 and the following
apply.
4.1 Biometric data
4.1.1
biometric reference
〈template, model〉 user’s stored reference measure based on features extracted from enrolment samples
4.2 Components of a biometric system
4.2.1
feature extractor
apparatus that extracts features from a sample
4.2.2
biometric reference generator
apparatus that transforms a sample into a biometric reference
4.3 User interaction with a biometric system
4.3.1
acclimatization
reduction, over the course of an evaluation, in a temporal condition of a biometric characteristic that may
impact the ability of a sensor to process a sample
4.3.2
effort level
number of presentations, attempts or transactions needed to successfully enrol or match in a biometric system
4.3.3
enrolment attempt
submission of one or more biometric samples for a Test Subject for the purpose of enrolment in a biometric
system
NOTE 1 One or more enrolment attempts may be permitted or required to constitute an enrolment transaction. An
enrolment attempt may comprise one or more enrolment presentations.
NOTE 2 See Annex B for illustration of the relationship between presentation, attempt and transaction.
4.3.4
enrolment attempt limit
maximum number of attempts, or the maximum duration, a Test Subject is permitted before an enrolment
transaction is terminated
4.3.5
enrolment presentation
submission of an instance of a biometric characteristic for a Test Subject for the purpose of enrolment
NOTE One or more enrolment presentations may be permitted or required to constitute an enrolment attempt. An
enrolment presentation may or may not result in an enrolment attempt.
4.3.6
enrolment presentation limit
maximum number of presentations, or the maximum duration, a Test Subject is permitted before an enrolment
attempt is terminated
2 © ISO/IEC 2007 – All rights reserved
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.