ISO/IEC 19795-4:2008
(Main)Information technology — Biometric performance testing and reporting — Part 4: Interoperability performance testing
Information technology — Biometric performance testing and reporting — Part 4: Interoperability performance testing
ISO/IEC 19795-4:2008 prescribes methods for technology and scenario evaluations of multi-supplier biometric systems that use biometric data conforming to biometric data interchange format standards. It specifies requirements needed to assess performance available from samples formatted according to a standard interchange format (SIF), performance available when samples formatted according to a SIF are exchanged, performance available from samples formatted according to a SIF, relative to proprietary data formats, SIF interoperability by quantifying cross-product performance relative to single-product performance, performance available from multi-sample and multimodal data formatted according to one or more SIFs, and performance interoperability of biometric capture devices. In addition, ISO/IEC 19795-4:2008 includes procedures for establishing an interoperable set of implementations, defines procedures for testing interoperability with previously established sets of implementations, and gives testing procedures for the measurement of interoperable performance. It does not establish a conformance test for biometric data interchange formats, or provide test procedures for online data collection.
Technologies de l'information — Essais et rapports de performances biométriques — Partie 4: Essais de performances d'interopérabilité
General Information
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 19795-4
First edition
2008-06-01
Information technology — Biometric
performance testing and reporting —
Part 4:
Interoperability performance testing
Technologies de l'information — Essais et rapports de performances
biométriques —
Partie 4: Essais de performances d'interopérabilité
Reference number
ISO/IEC 19795-4:2008(E)
©
ISO/IEC 2008
---------------------- Page: 1 ----------------------
ISO/IEC 19795-4:2008(E)
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2008
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2008 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 19795-4:2008(E)
Contents Page
Foreword .vi
Introduction.vii
1 Scope.1
2 Conformance .1
3 Normative references.2
4 Terms and definitions .2
5 Abbreviated terms .4
6 Goals.5
6.1 Coverage .5
6.2 Target application.8
6.2.1 Biometric application.8
6.2.2 Interoperable application.9
6.3 Purpose .10
6.3.1 Interoperability testing.10
6.3.2 Sufficiency testing.11
7 Metrics .12
7.1 General .12
7.2 Figures of merit .12
7.2.1 Recognition performance figure of merit.12
7.2.2 Measuring component failure .13
7.3 Interoperability matrices.14
7.3.1 General .14
7.3.2 Interoperability with sBDB generators.14
7.3.3 Interoperability with sBDB generators.15
7.3.4 Fixed operating point interoperability.16
7.3.5 Reporting failure of sBDB generators.16
7.4 Proprietary performance.16
8 Conducting a test .17
8.1 Structure of test.17
8.2 Sample data .17
8.2.1 Acquisition .17
8.2.2 Representative data .18
8.2.3 Collection of ancillary data.18
8.2.4 Corpus size .18
8.2.5 Removal of subject-specific metadata .18
8.2.6 Removal of unrepresentative metadata .18
8.2.7 Origin of samples .19
8.2.8 Untainted samples.19
8.2.9 Sequestered data.19
8.3 Conformance testing.19
8.3.1 Conformance .19
8.3.2 Executing conformance tests .19
8.3.3 Reporting.20
8.4 Constraints on the sBDBs.20
8.4.1 Optional encodings .20
8.4.2 Optional encodings from profile standards.20
8.4.3 Deviation from the base standard .20
8.4.4 Data encapsulation.20
© ISO/IEC 2008 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 19795-4:2008(E)
8.5 Components.21
8.5.1 Components for sufficiency testing .21
8.5.2 Establishing modularity requirements .21
8.5.3 Components for interoperability testing .21
8.5.4 Underlying algorithms.21
8.5.5 Capture device user interfaces .21
8.5.6 Multimodal components .22
8.5.7 Component variability .22
8.5.8 Component reporting requirements .22
8.6 Planning decisions .22
8.6.1 Computational intensity.22
8.6.2 Supplier recruitment.23
8.6.3 Provision of samples to suppliers .23
8.6.4 Equivalency of generator resources.23
8.6.5 Handling violations of test requirements.24
8.6.6 Comparison subsystem output data encapsulation.24
8.6.7 Fundamental generator requirement.24
8.6.8 Fundamental comparison subsystem requirement .25
8.6.9 General requirements on software implementations.25
8.7 Prevention and detection of gaming.26
8.7.1 General aspects .26
8.7.2 Modes of gaming .26
8.7.3 Prevention and detection of gaming.28
8.8 Test procedure.29
8.8.1 Primary test .29
8.8.2 Uncertainty measurement.30
8.8.3 Variance estimation.30
8.8.4 Remedial testing .30
8.8.5 Survey of configurable parameters .30
9 Interpretation of the interoperability matrix.30
9.1 Determination of interoperable subsystems .30
9.1.1 General.30
9.1.2 Identifying interoperable combinations of subsystems.31
9.1.3 Acceptable numbers of interoperable subsystems .33
9.1.4 Combinatorial search for maximum interoperability-classes.33
9.1.5 Multiple interoperable subgroups.34
9.1.6 Statistical stability of the test result .34
9.2 Interoperability with previously certified products.35
9.2.1 Decertification considerations .35
9.2.2 Continuity of testing.35
9.2.3 Interoperability with previously certified generators.35
9.2.4 Interoperability with previously certified comparison subsystems.36
9.2.5 Treatment of systematic effects.36
9.2.6 Retroactive exclusion from analysis .37
9.3 Overall sufficiency.37
Annex A (informative) Procedures for conducting a test of sufficiency and/or interoperability.38
Annex B (informative) Example Interoperability Test.42
Bibliography .45
Figure 1 — General biometric interoperability .6
Figure 2 — Specific interoperability: enrolment BDB is standardized .6
Figure 3 — Specific interoperability: enrolment BDB is proprietary.7
Figure 4 — Offline interoperability testing.7
Figure 5 — Biometric capture device interoperability .8
iv © ISO/IEC 2008 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 19795-4:2008(E)
Figure 6 — Cells of an example interoperability space.10
Figure 7 — Sufficiency testing: proprietary vs. standard interchange formats .12
Figure 8 — Cross-generator performance matrix .15
Figure 9 — Example performance matrix .15
Figure 10 — Proprietary performance matrix.16
Table 1 – Conformity with ISO/IEC 19795-2 .1
Table 2 – Sample size adjustment of error rate requirement.31
Table 3 – Confidence levels of the standard Normal distribution.32
Table A.1 – Interoperability test procedure, phase 1: planning.38
Table A.2 – Interoperability test procedure, phase 2: setup.39
Table A.3 – Interoperability test procedure, phase 3: sBDB and pBDB generation.39
Table A.4 – Interoperability test procedure, phase 4: verification .40
Table A.5 – Interoperability test procedure, phase 5: identification .40
Table A.6 – Interoperability test procedure, phase 6: reporting.41
Table A.7 – Interoperability test procedure, phase 7: variance estimation.41
© ISO/IEC 2008 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 19795-4:2008(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 19795-4 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 37, Biometrics.
ISO/IEC 19795 consists of the following parts, under the general title Information technology — Biometric
performance testing and reporting:
⎯ Part 1: Principles and framework
⎯ Part 2: Testing methodologies for technology and scenario evaluation
⎯ Part 3: Modality-specific testing [Technical Report]
⎯ Part 4: Interoperability performance testing
Part 6: Testing methodologies for operational evaluation is under preparation.
vi © ISO/IEC 2008 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC 19795-4:2008(E)
Introduction
The multi-part biometric data interchange format standard, ISO/IEC 19794, has been developed to foster
interoperable exchange of biometric data. By defining open containers for image, signal and feature data, and
constraining some of the properties of the samples, the standards enhance interoperability by requiring
implementers to be able to handle a restricted set of all possible biometric samples. Examples of this are the
template standards of ISO/IEC 19794-2 and ISO/IEC 19794-8 which embed compact processed data from
fingerprint images. Only samples of the same format type (several of which can be defined in the same part of
ISO/IEC 19794) are intended to be interchangeable.
One common assertion prior to SC 37's formulation of data interchange standards was that proprietary
templates offer greater recognition performance than any likely standard on the grounds that the proprietary
instances are the product of processes that embed considerable, private, intellectual property. The question of
whether the emerging standards are sufficient then arises: that is, do they code data (feature, image, etc.)
representations that allow matching with accuracy comparable to that available from the proprietary solutions?
A second issue, interoperability, arises in those applications where standardized data are generated and
matched by different institutions and systems. If a company’s feature extraction subsystem processes
acquired samples to produce ISO/IEC 19794-x compliant instances, then can other companies' comparison
subsystems attain performance comparable with that obtained from the originator's own comparison
subsystem? A further question is then whether a third company can successfully recognize enrolment and
user samples from two different sources.
This part of ISO/IEC 19795 defines tests to specifically address absolute performance, sufficiency, and
interoperability available from biometric data formatted to comply with established standards, particularly
those developed in the various parts of ISO/IEC 19794. However, because this part of ISO/IEC 19795
references interchange formats generically, by referencing only their black box generation and use, it also
applies to other open standards. One consequence of this approach is that the success of a test is predicated
on the correctness and appropriateness of lower-level data elements and values, i.e. conformance to the
respective standards. Therefore, the approach here is to require conformance testing as an integral part of the
test. This is achieved by referencing formal published conformance tests or profiles of standards. For instance,
an interoperability test of the ISO/IEC 19794-5 face format might reference an application profile of its Token
image, which in turn might rely on ISO/IEC 15444-1 (JPEG 2000 core coding system).
This part of ISO/IEC 19795 conceives of the following three kinds of tests:
⎯ online: a scenario test in which a volunteer population enrols on suppliers' products and
subsequently uses suppliers' verification or identification implementations to make genuine and
impostor attempts;
⎯ offline: a technology test in which an archived corpus of captured samples, not necessarily collected
with any intent to simulate the operational conditions of a particular application, is used as input to
suppliers' enrolment, verification or identification products to make genuine and impostor attempts;
⎯ hybrid: a test in which the sample corpus is collected online under conditions which attempt to
simulate the operational conditions of a particular application, and is then processed offline.
In each case, an interoperability test needs to embed multi-supplier generation, exchange, and comparison of
samples of the standard interchange format. Online collection from a live population is appropriate when the
biometric capture device, and/or the subject interaction with the biometric capture device, is considered to
have a material effect on the interoperable performance of the intended application. An offline test is
appropriate when a representative corpus of samples is already available (for example passport photographs
to be converted into Token instances of ISO/IEC 19794-5). An offline test may be appropriate when the
collection of representative data is neither practical nor necessary to determine the interoperable performance
of specific subsystems, such as feature extraction and/or comparison.
© ISO/IEC 2008 – All rights reserved vii
---------------------- Page: 7 ----------------------
ISO/IEC 19795-4:2008(E)
In all cases, an interoperability test must enrol subjects on one or more products and verify or identify on one
or more others. This should involve subjects making transactions as themselves (genuine trials) and as one or
more other people (impostor trials). If a large enough population is available, a disjoint impostor population
can be used. Since online tests can become onerous on the test population when many products and
impostor attempts are needed, hybrid and offline testing allow execution of many zero-effort impostor attempts.
In an interoperability performance test, J generators of standardized biometric data blocks (BDBs) are applied
to the samples assembled as part of a hybrid or offline test. By applying K comparison subsystems to the
2
standard BDBs, up to KJ verification or identification trials are conducted, each following ISO/IEC 19795-2.
The BDB may be an image or signal, or a standardized template. Optional encodings allowed by the standard
interchange format should be fully specified. This might be achieved by normatively referencing one of the
ISO/IEC 24713-x profiles. If the format in question is an image, a subsequent internal (usually proprietary)
template would be used, but its existence here is subsumed by the notion of a black-box comparison of two
instances of the given format.
The test advanced by this part of ISO/IEC 19795 demarcates the generic aspects of interoperability from the
meaning associated with each particular biometric format of ISO/IEC 19794-x.
viii © ISO/IEC 2008 – All rights reserved
---------------------- Page: 8 ----------------------
INTERNATIONAL STANDARD ISO/IEC 19795-4:2008(E)
Information technology — Biometric performance testing and
reporting —
Part 4:
Interoperability performance testing
1 Scope
This part of ISO/IEC 19795 prescribes methods for technology and scenario evaluations of multi-supplier
biometric systems that use biometric data conforming to biometric data interchange format standards.
It specifies requirements needed to assess
⎯ performance available from samples formatted according to a standard interchange format (SIF),
⎯ performance available when samples formatted according to a SIF are exchanged,
⎯ performance available from samples formatted according to a SIF, relative to proprietary data formats,
⎯ SIF interoperability, by quantifying cross-product performance relative to single-product performance,
⎯ performance available from multi-sample and multimodal data formatted according to one or more SIFs,
and
⎯ performance interoperability of biometric capture devices.
In addition, this part of ISO/IEC 19795
⎯ includes procedures for establishing an interoperable set of implementations,
⎯ defines procedures for testing interoperability with previously established sets of implementations, and
⎯ gives testing procedures for the measurement of interoperable performance.
It does not
⎯ establish a conformance test for biometric data interchange formats, or
⎯ provide test procedures for online data collection.
2 Conformance
An interoperability performance test conforms to this part of ISO/IEC 19795 if it satisfies the requirements
specified in Clauses 6, 7, 8 and 9 of this part of ISO/IEC 19795 and the requirements specified in the clauses
of ISO/IEC 19795-2 referenced in Table 1.
Table 1 — Conformity with ISO/IEC 19795-2
Structure of ISO/IEC 19795-4 test ISO/IEC 19795-2 conformance
Online (8.2.1.3) Clause 7 (Scenario evaluation)
Hybrid (8.2.1.4) Clause 6 and Clause 7
Offline (8.2.1.2) Clause 6 (Technology evaluation)
© ISO/IEC 2008 – All rights reserved 1
---------------------- Page: 9 ----------------------
ISO/IEC 19795-4:2008(E)
3 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition o
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.