ISO/IEC 19794-1:2011/Amd 1:2013
(Amendment)Information technology — Biometric data interchange formats — Part 1: Framework — Amendment 1: Conformance testing methodology
Information technology — Biometric data interchange formats — Part 1: Framework — Amendment 1: Conformance testing methodology
Technologies de l'information — Formats d'échange de données biométriques — Partie 1: Cadre — Amendement 1: Méthodologie d'essai de conformité
General Information
Relations
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 19794-1
Second edition
2011-07-15
AMENDMENT 1
2013-02-01
Information technology — Biometric data
interchange formats —
Part 1:
Framework
AMENDMENT 1: Conformance testing
methodology
Technologies de l'information — Formats d'échange de données
biométriques —
Partie 1: Cadre
AMENDEMENT 1: Méthodologie d'essai de conformité
Reference number
ISO/IEC 19794-1:2011/Amd.1:2013(E)
©
ISO/IEC 2013
---------------------- Page: 1 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2013
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2013 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees established
by the respective organization to deal with particular fields of technical activity. ISO and IEC technical
committees collaborate in fields of mutual interest. Other international organizations, governmental and non-
governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology,
ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
Amendment 1 to ISO/IEC 19794-1:2011 was prepared by Joint Technical Committee ISO/IEC JTC 1,
Information technology, Subcommittee SC 37, Biometrics.
© ISO/IEC 2013 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
Information technology — Biometric data interchange
formats —
Part 1:
Framework
AMENDMENT 1: Conformance testing methodology
Page vi, Introduction
Add the following paragraphs after Figure 1:
Users of biometric systems desire to use this and other standards to ensure that components of the
biometric system can be substituted with other components from different vendors with a minimum of
effort, and also to ensure that biometric data produced by one system can be used by another system. In
order to achieve this, it is critical that systems claimed to conform to a standard actually are conformant,
and thus there is a need for a standardized conformance testing methodology for each of the biometric
data interchange formats described in ISO/IEC 19794, in order to provide a reasonable degree of
assurance that a conformance claim has validity. While conformance of individual elements of data
interchange records to relevant requirements can be determined, no test can be absolutely
comprehensive and prove that a given system generating or using biometric data interchange records is
conformant under all possible circumstances, especially when there are optional components of the
standard. A well designed conformance test can, however, test all of the most likely sources of problems
and ensure that the implementation under test conforms under a reasonable set of circumstances, giving
assurance, but not a guarantee, of conformance.
There are many different types of conformance testing that may be appropriate for the various parts of
ISO/IEC 19794. Some of these tests are highly specific to each data interchange format but some of them
have many common elements across all of the formats. This part of ISO/IEC 19794 also describes the
different types of conformance testing, and provides details of the common elements for defining test
assertions. It also provides guidelines for conducting the tests and reporting the results of the tests. Tests
and assertions common for most or all biometric data interchange formats (e.g., for common elements of
the general headers and the common elements of the representation headers) are specified in this part of
ISO/IEC 19794; the specific tests and assertions for each biometric data interchange format are left to the
subsequent parts of ISO/IEC 19794.
Annex A of this part of ISO/IEC 19794 is distinct from the ISO/IEC 29109-1 which addressed
conformance testing only of the first edition of ISO/IEC 19794. The normative Annex A of this part of
ISO/IEC 19794 addresses conformance testing of data formats specified in the second edition of ISO/IEC
19794.
Page 1, Scope
Add the following text at the end of the Scope:
This part of ISO/IEC 19794 also specifies the concepts, test types and conformance testing
methodologies to test biometric data interchange records or computer algorithms that create biometric
data interchange records. It defines two types (type A, i.e., biometric data interchange records and
systems generating such records; and type B, i.e., systems using biometric data interchange records),
and three levels (Level1, i.e., checking internal content of each field; Level 2, i.e., checking internal
consistency of the entire record; and Level 3, i.e., checking whether the data record is a faithful
© ISO/IEC 2013 – All rights reserved 1
---------------------- Page: 4 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
representation of the original biometric data) of conformance testing, but it only provides a detailed
description and methodology for the three levels of Type A testing. This part of ISO/IEC 19794 specifies
test requirements, assertions, and test execution and reporting procedures that are common for most or
all biometric data interchange formats. It explicitly does not cover the following areas:
• Modality-specific detailed test elements and assertions or descriptions of any mandatory standard
datasets required for testing. They are provided in the subsequent parts of ISO/IEC 19794.
• Testing whether implementations under test (IUTs) that claim to be able to use conformant biometric
data interchange records are able to correctly process such biometric data interchange records (Type B
testing).
• Conformance testing of CBEFF requirements
• Testing of other characteristics of biometric products or other types of testing of biometric products
(i.e., acceptance, performance, robustness, security).
Page 1
Add the following clause after the Scope and renumber all subsequent clauses accordingly:
2 Conformance
Biometric data interchange format conformance tests that claim conformance to this part of ISO/IEC
19794 shall satisfy the normative requirements of the methodology for those levels of test they are
claiming to perform, as described in Clauses A.1, A.2 and A.3. Any conformance tests shall use the
assertion types defined in Clause A.2 with the specific assertion details given in this and the relevant
subsequent parts of this standard.
Implementations of subsequent parts of ISO/IEC 19794 tested according to the methodology specified in
this part of ISO/IEC 19794 shall be able to claim conformance only to those requirements specified in
ISO/IEC 19794 that are tested by the test methods established by this methodology.
Pages 1 to 8, Terms and definitions
Add the following terms and definitions alphabetically , renumbering accordingly:
4.x
assertion
specification for testing a conformance requirement in an implementation under test expressed in a formal
assertion definition language
4.x
assertion test
specification of software or procedural methods that generate the test outcomes used for assessment of
conformance to an assertion
NOTE This is adapted from the definition of “assertion test” in ISO/IEC 13210:1999.
4.x
attestation
issue of a statement, based on a decision that fulfillment of specified requirements has been
demonstrated
NOTE This is adapted from the definition of “attestation” in ISO/IEC 17000:2004.
2
© ISO/IEC 2013 – All rights reserved
---------------------- Page: 5 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
4.x
certification
third-party attestation related to products
[ISO/IEC 17000:2004]
4.x
conformance
conformity
fulfillment by a product, process, or service of all relevant specified conformance requirements
NOTE For all practical purposes, data records are considered to be a type of a “product”, i.e., provisions of this
standard that are applicable to “products” apply to data records.
4.x
conformance requirement
requirement stated in a data format specification and defined in a finite, measurable, and unambiguous
manner
NOTE This is adapted from the definition of “conformance requirement” in ISO/IEC 13210:1999.
4.x
conformance test
specified technical procedure of conformance testing
4.x
conformance testing
testing
determination of one or more characteristics of an object of conformity assessment, according to a
procedure
[ISO/IEC 17000:2004]
4.x
conformance testing laboratory
organization that carries out conformance testing.
EXAMPLE This may be the creator of the IUT, the user of the IUT, or an unbiased third party.
4.x
conformance test specification
test specification
provisions of ISO/IEC 19794 biometric data interchange format standard that is concerned with test
methods, sometimes supplemented with other provisions related to testing, such as sampling, use of
statistical methods, and sequence of tests
NOTE This is adapted from the definition of “testing standard” in ISO/IEC GUIDE 2:2004.
4.x
conformance testing suite
CTS
test software used to automate certain types of conformance testing
4.x
conformity assessment
demonstration that specified requirements relating to a product, process, system, person or body are
fulfilled
[ISO/IEC 17000:2004]
© ISO/IEC 2013 – All rights reserved 3
---------------------- Page: 6 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
4.x
data format specification
provisions of ISO/IEC 19794 biometric data interchange format standard containing the specification that
is the subject of the conformance testing
4.x
declaration
declaration of conformity
first-party attestation
[ISO/IEC 17000:2004]
4.x
Implementation conformance statement
ICS
statement by the supplier of an implementation under test that indicates which mandatory and optional
components of the data format specification are supported by the implementation
4.x
implementation under test
IUT
that which implements the data format specification being tested
NOTE Depending on the conformance requirements of the data format specification, this may simply be a set of
biometric data interchange records or it may be a computer algorithm in the form that creates the BDIR and/or uses
the data contained in the BDIR.
4.x
input biometric data record
IBDR
data package containing a less processed form of biometric data which is suitable for use in the creation
of a BDIR
NOTE In some cases, this may be an image, but it may also be raw sensor output such as a time series of data
points from a digitization tablet.
4.x
level 1 testing
conformance testing methodology that checks field by field and byte by byte conformance with the
specification of the BDIR as specified in the data format specification, both in terms of fields included and
the ranges of the values in those fields
NOTE This type of testing tests syntactic requirements of the data format specification.
4.x
level 2 testing
conformance testing methodology that tests the internal consistency of the BDIR under test, relating
values from one part or field of the BDIR to values from other parts or fields of the BDIR
NOTE This type of testing tests syntactic requirements of the data format specification.
4.x
level 3 testing
conformance testing methodology that tests that a BDIR produced by an IUT is a faithful representation of
the IBDR subject to the constraints of the parameters in the metadata records
NOTE This type of testing tests semantic requirements of the data format specification.
4
© ISO/IEC 2013 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
4.x
metadata record
data record containing any specific parameters related to the data itself, particularly required by an IUT to
transform an IBDR into a BDIR
EXAMPLE Type of image (basic, full frontal, token frontal or other) and the level of compression for a face image
BDIR; the presence of core, delta, or ridgecounts in the extended area for finger minutiae BDIR; the size of each
pattern in a finger pattern BDIR.
4.x
procedure
specified way to carry out an activity or a process
[ISO 9000:2005]
4.x
requirement
provision that conveys criteria to be fulfilled
[ISO/IEC GUIDE 2:2004]
4.x
test method
specified technical procedure for performing a test
NOTE This is adapted from the definition of “test method” in ISO/IEC GUIDE 2:1996. More recent edition of
ISO/IEC GUIDE 2 or ISO/IEC 17000:2004 no longer contain this definition
4.x
test report
document that presents test results and other information relevant to the execution of the test methods
against an Implementation Under Test
NOTE This is adapted from the definition of “test report” in ISO/IEC 13210:1999 and ISO/IEC GUIDE 2:1996.
4.x
Type A conformance claim
conformance claim that an IUT is a conformant BDIR, or can create conformant BDIRs from appropriate
IBDRs
4.x
Type B conformance claim
conformance claim that an IUT can read conformant BDIRs, interpret them correctly, and perform its
desired function upon them
Page 8, Abbrevaited terms
Add the following abbreviated terms alphabetically:
CTS Conformance Testing Suite
ICS Implementation Conformance Statement
IUT Implementation Under Test
IBDR Input Biometric Data Record
© ISO/IEC 2013 – All rights reserved 5
---------------------- Page: 8 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
Page 14, 8.1 (now renumbered as 9.1)
In the last sentence, change "Annex A" to "Annex B".
Page 25, Annex A
Rename Annex A as Annex B.
Page 24
After Table 7, insert the following annex:
6
© ISO/IEC 2013 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
Annex A
(normative)
Conformance testing methodology
A.1 Conformance testing framework
A.1.1 Limitations
While conformance of individual elements of each data interchange record to relevant requirements can be
determined, no conformance test of a given system generating or using biometric data interchange records
can be complete or perfect. Ultimately, it is only possible to prove that an IUT is non-conformant. The goal of
conformance testing is therefore to capture enough of the requirements of the data format specification and
test them under enough conditions, that any IUT that passes the conformance test is likely to be conformant.
Two problems with a data format specification that may only become apparent during conformance testing are
that some areas may be undefined (so that the specification of these areas is left to each vendor) or ill-defined
(so that there is a contradiction between parts of the data format specification or an easy misinterpretation
caused by the wording of the data format specification). The latter problem may be resolved by an
amendment to the standard, but the former problem may be difficult to resolve. An obvious example is the use
of proprietary extended data blocks within a BDIR. There may be good reasons to allow such proprietary data,
but very little conformance testing is possible while the data remains proprietary. Also, if the data format
specification includes a requirement to interpret the BDIR or use it for biometric comparison, then it is difficult
to be sure of the effect of a proprietary data block produced by one IUT when another IUT is attempting to
interpret it.
A.1.2 Managing data records
Note that since CBEFF conformance testing is out of scope for this standard, it is generally assumed that the
BDIRs will have been removed from any CBEFF data structures prior to beginning the conformance test.
Typically, for Type A testing as described in Clause A.1.3, either the IUT will provide BDIRs without a CBEFF
encapsulator or the CTS will remove them from such an encapsulator if one exists. Regardless of the method
used, the test shall provide a means of passing the CBEFF format type corresponding to the IBDRs in the IUT
or produced by the IUT to the CTS. This may be as simple as the supplier of the IUT sending a written
instruction to the testing laboratory that all BDIRs produced by this IUT would have a particular format type, or
it may involve the IUT passing a special parameter or using a specific CBEFF patron format that is not part of
its normal function outside the test. The reason this is required is that several parts of ISO/IEC 19794 have
different format types that determine whether or not certain optional data is present. Thus format type is an
extra field that shall always be present together with a BDIR when conformance testing using that BDIR
occurs.
A.1.3 Conformance testing types
Generally, the goal of biometric data interchange format conformance testing is to assure the users of
conformant biometric products that a BDIR produced by any conformant product can be interpreted and used
correctly by any other conformant product. There are thus two types of fundamental conformance claims.
Type A is the ability to produce conformant BDIRs and Type B is the ability to use conformant BDIRs.
Different IUTs may have different purposes for which they use a conformant BDIR, and thus Type B testing is
more complex than Type A testing because it has to account for all of these purposes. ISO/IEC 19794 is
focused on Type A testing exclusively. When biometric data interchange records themselves are tested in the
absence of any software or hardware that produced them or uses them, this is treated as Type A testing.
© ISO/IEC 2013 – All rights reserved 7
---------------------- Page: 10 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
A.1.4 Conformance testing levels
A.1.4.1 Hierarchy of Conformance Tests
A first step towards the goal of demonstrating conformance is ensuring that all of the specified fields and data
structures in the BDIR are correct and self-consistent. This does not validate the fidelity of the information
contained in the BDIR, however, since that depends on the relationship between the original IBDR and the
BDIR. This leads to a natural hierarchy of conformance testing levels.
The conformance testing hierarchy presented in this standard has three levels. Generally they progress from
least complex and expensive to test to most complex and expensive to test. They also progress from less
useful in predicting the performance of real world systems using conformant products, to more useful,
although even Level 1 conformance testing represents a significant step towards that goal. The types of
assertions for Level 1 and Level 2 testing for all parts of ISO/IEC 19794 are similar and so a list of assertion
operators and operands that should be used to define assertions is given in Clause A.2 of this part of
ISO/IEC 19794. The details of all the Level 1 and Level 2 assertions for each data format specification are
given in the appropriate subsequent parts of ISO/IEC 19794. For the more complex Level 3 testing, where the
actual fidelity of the information in the BDIR is compared to that in the IBDR, the subsequent parts of
ISO/IEC 19794 provide, as far as it is possible, guidance on how to carry out Level 3 testing for their specific
data interchange formats. A given conformance test may therefore involve conformance testing at different
levels.
It will ultimately be up to application profiles or to individual users of ISO/IEC 19794 to determine which level
of conformance testing will be required for a specific application, as well as any requirements on performance
or interoperability. This will be dependent on time, cost, and importance of biometric performance, implications
of non-interoperability and the current state of the published versions of the various parts of ISO/IEC 19794.
A.1.4.2 Level 1 – Data format conformance
In Level 1 testing, a set of BDIRs shall be checked for field-by-field and byte-by-byte conformance with the
specification of the data format specification, both in terms of fields included and the ranges of the values in
those fields. The specific assertions tested for each data format specification shall be those described in the
appropriate part of ISO/IEC 19794.
The advantage of this testing is that it does not require an IUT to be a computer algorithm or a set of hardware
and software. It can simply be a set of BDIRs. Thus, any hardware or software components of the
implementation being tested do not have to come into the possession of the testing lab, only BDIRs created
with those components.
An IUT may have the capability to produce multiple BDIRs, depending on the requirements of the application
in which it is used. Some of these BDIRs may be conformant and others may not, and so it is important to
specify which types were tested and how many of each type. In an ideal world every possible combination of
parameters for a particular biometric data interchange format would be tested, but this is not realistic given the
resources that would be required for such testing. Provided a test reports the presence or absence of optional
fields and the values for variable structural fields it is possible for a user of the data format specification to
determine if the particular variant of the standard tested is appropriate for their needs. The user may also
require conformance test results for a specific type of BDIR. An obvious example would be a two finger BDIR
or two iris BDIR, since many applications require enrollment of more than one biometric characteristic in order
to allow for a back-up if one of them becomes damaged or temporarily unusable. Some IUTs might be
conformant with a single-representation BDIR, or even with multiple representations of a single finger or iris,
but might fail conformance testing when the BDIR contains more than one finger or iris.
A.1.4.3 Level 2 – Internal consistency checking
In Level 2 testing, a set of BDIRs shall be checked to determine if they are internally consistent. The specific
assertions tested for each data format specification shall be those described in the appropriate part of
ISO/IEC 19794.
8
© ISO/IEC 2013 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
The nature of Level 2 testing is that it relates values from one part of the BDIR to values from other parts of
the BDIR. This may be due to explicit requirements in the data format specification, such as a requirement
that the record length actually does indicate the number of bytes in the BDIR. It may also be implicit in the
standard, such as determining that the coordinates of a particular feature (such as eye positions in a face
image record or minutiae positions in a finger minutiae record) actually fall within the specified size of the
image.
In some cases, test assertions for Level 2 and higher conformance testing will have to make specific
assumptions about interpreting the data format specification requirements. In those cases, testing
methodology as set forth in ISO/IEC 19794 shall be considered normative in its interpretation of the data
format specification requirements, and any other interpretations shall be considered non-conformant to the
data format specification.
Once again the advantage of this testing is that it does not require an IUT to be a computer algorithm or a set
of hardware and software. It can simply be a set of BDIRs. Then the hardware or software of the IUT does not
have to be part of the test, only BDIRs created with that implementation. The disadvantage is that there are a
limited number of BDIRs and it is quite possible that some of the internal consistency checks will never be
tested because they are not relevant for the set of BDIRs in the IUT. The solution is to test a larger number of
BDIRs that represent multiple different structural variants of the biometric data interchange format under test.
This is why it is so vital to report on the structure of each BDIR variant in the conformance test.
Since Level 1 and Level 2 conformance testing are both required in order to properly test that the structure of
a BDIR is conformant to a data format specification, and since the execution of Level 1 and Level 2 tests will
frequently be intermingled, a conformance test should always include all relevant Level 1 and Level 2 test
assertions.
A.1.4.4 Level 3 – Content checking
Level 3 conformance testing is defined as a conformance testing methodology that tests that a BDIR produced
by an IUT is a faithful representation of the IBDR subject to the constraints of the parameters in the metadata
records. Effectively this is intended to test that the BDIRs produced by an IUT are faithful representations of
the original biometric data and that they satisfy those requirements of the data format specification that are not
simply a matter of syntax and format. In some cases the requirements of the data format specification may
specify biometric data capture conditions. An example would be the use of a fingerprint sensor of a particular
resolution or having a particular certification by an external body to capture fingerprints and generate
fingerprint image records that are noted in the BDIR as having a particular image acquisition level. The only
way to test that such a requirement has been correctly implemented by the IUT is to require that in Level 3
testing, an IUT shall be a combination of computer hardware and/or software that is used in the testing
laboratory. If the IUT is software only, then a set of IBDRs and corresponding metadata records shall be
provided and the IUT shall produce a set of corresponding BDIRs. This, however, only tests the ability of the
IUT to parse the metadata records and the IBDRs and insert the appropriate information in correctly formatted
BDIRs. For some requirements it is essential that the entire process from data capture to BDIR production be
included in the test, and in those cases the IUT will have to be a complete set of hardware and software.
Finally, there are some requirements that cannot be quantitatively tested without significant special effort or
extra equipment and for which the conformance test specification may simply define that no Level 3 test is
possible. An example would be the pose angles listed in a face image data record. Without an external three
dimensional reference frame for the head containing the face that is represented in the data record, there is
no absolute mechanism to verify that these pose angles are correct.
This leads to the following methodology for handling Level 3 conformance tests. As indicated in Clause A.3, all
of the requirements of the data format specification shall be listed in a conformance requirements table that is
in the same form as an implementation conformance statement. This will help the supplier of the IUT to clearly
identify which requirements of the data format specification are supported by the implementation. Certain
columns in this table indicate whether each requirement is a Level 1, Level 2 or Level 3 requirement and for
each Level 3 requirement, indicate whether it can be tested using a software only solution with a database of
IBDRs and metadata records, whether it requires a complete hardware and software IUT, or whether it can’t
be tested at all without special effort. Where they are available, the specific test methodologies to be used for
Level 3 testing involving IUTs composed of both hardware and software are found within specific clauses, as
referenced in the table, in each of the subsequent parts of ISO/IEC 19794. A general methodology to test
© ISO/IEC 2013 – All rights reserved 9
---------------------- Page: 12 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
some Level 3 requirements using a software only solution and a database of IBDRs and metadata records is
described in this clause, but even it will require specific clauses in the subsequent parts of ISO/IEC 19794 on
how to use this methodology for specific requirements.
The basic structure of a software only Level 3 conformance test is that a set of IBDRs and corresponding
metadata records shall be provided and the IUT shall produce a set of corresponding BDIRs. The information
in the BDIRs shall then be compared to the information in the IBDRs to determine if the IUT has faithfully
reproduced that in
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.