Information technology — Biometric data interchange formats — Part 1: Framework — Amendment 1: Conformance testing methodology

Technologies de l'information — Formats d'échange de données biométriques — Partie 1: Cadre — Amendement 1: Méthodologie d'essai de conformité

General Information

Status
Published
Publication Date
05-Feb-2013
Current Stage
6060 - International Standard published
Start Date
06-Feb-2013
Completion Date
06-Feb-2013
Ref Project

RELATIONS

Buy Standard

Standard
ISO/IEC 19794-1:2011/Amd 1:2013 - Conformance testing methodology
English language
29 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

INTERNATIONAL ISO/IEC
STANDARD 19794-1
Second edition
2011-07-15
AMENDMENT 1
2013-02-01
Information technology — Biometric data
interchange formats —
Part 1:
Framework
AMENDMENT 1: Conformance testing
methodology
Technologies de l'information — Formats d'échange de données
biométriques —
Partie 1: Cadre
AMENDEMENT 1: Méthodologie d'essai de conformité
Reference number
ISO/IEC 19794-1:2011/Amd.1:2013(E)
ISO/IEC 2013
---------------------- Page: 1 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2013

All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,

electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or

ISO's member body in the country of the requester.
ISO copyright office
Case postale 56  CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2013 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are members of

ISO or IEC participate in the development of International Standards through technical committees established

by the respective organization to deal with particular fields of technical activity. ISO and IEC technical

committees collaborate in fields of mutual interest. Other international organizations, governmental and non-

governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology,

ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.

International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.

The main task of the joint technical committee is to prepare International Standards. Draft International

Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as

an International Standard requires approval by at least 75 % of the national bodies casting a vote.

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent

rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.

Amendment 1 to ISO/IEC 19794-1:2011 was prepared by Joint Technical Committee ISO/IEC JTC 1,

Information technology, Subcommittee SC 37, Biometrics.
© ISO/IEC 2013 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
Information technology — Biometric data interchange
formats —
Part 1:
Framework
AMENDMENT 1: Conformance testing methodology
Page vi, Introduction
Add the following paragraphs after Figure 1:

Users of biometric systems desire to use this and other standards to ensure that components of the

biometric system can be substituted with other components from different vendors with a minimum of

effort, and also to ensure that biometric data produced by one system can be used by another system. In

order to achieve this, it is critical that systems claimed to conform to a standard actually are conformant,

and thus there is a need for a standardized conformance testing methodology for each of the biometric

data interchange formats described in ISO/IEC 19794, in order to provide a reasonable degree of

assurance that a conformance claim has validity. While conformance of individual elements of data

interchange records to relevant requirements can be determined, no test can be absolutely

comprehensive and prove that a given system generating or using biometric data interchange records is

conformant under all possible circumstances, especially when there are optional components of the

standard. A well designed conformance test can, however, test all of the most likely sources of problems

and ensure that the implementation under test conforms under a reasonable set of circumstances, giving

assurance, but not a guarantee, of conformance.

There are many different types of conformance testing that may be appropriate for the various parts of

ISO/IEC 19794. Some of these tests are highly specific to each data interchange format but some of them

have many common elements across all of the formats. This part of ISO/IEC 19794 also describes the

different types of conformance testing, and provides details of the common elements for defining test

assertions. It also provides guidelines for conducting the tests and reporting the results of the tests. Tests

and assertions common for most or all biometric data interchange formats (e.g., for common elements of

the general headers and the common elements of the representation headers) are specified in this part of

ISO/IEC 19794; the specific tests and assertions for each biometric data interchange format are left to the

subsequent parts of ISO/IEC 19794.

Annex A of this part of ISO/IEC 19794 is distinct from the ISO/IEC 29109-1 which addressed

conformance testing only of the first edition of ISO/IEC 19794. The normative Annex A of this part of

ISO/IEC 19794 addresses conformance testing of data formats specified in the second edition of ISO/IEC

19794.
Page 1, Scope
Add the following text at the end of the Scope:

This part of ISO/IEC 19794 also specifies the concepts, test types and conformance testing

methodologies to test biometric data interchange records or computer algorithms that create biometric

data interchange records. It defines two types (type A, i.e., biometric data interchange records and

systems generating such records; and type B, i.e., systems using biometric data interchange records),

and three levels (Level1, i.e., checking internal content of each field; Level 2, i.e., checking internal

consistency of the entire record; and Level 3, i.e., checking whether the data record is a faithful

© ISO/IEC 2013 – All rights reserved 1
---------------------- Page: 4 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)

representation of the original biometric data) of conformance testing, but it only provides a detailed

description and methodology for the three levels of Type A testing. This part of ISO/IEC 19794 specifies

test requirements, assertions, and test execution and reporting procedures that are common for most or

all biometric data interchange formats. It explicitly does not cover the following areas:

• Modality-specific detailed test elements and assertions or descriptions of any mandatory standard

datasets required for testing. They are provided in the subsequent parts of ISO/IEC 19794.

• Testing whether implementations under test (IUTs) that claim to be able to use conformant biometric

data interchange records are able to correctly process such biometric data interchange records (Type B

testing).
• Conformance testing of CBEFF requirements

• Testing of other characteristics of biometric products or other types of testing of biometric products

(i.e., acceptance, performance, robustness, security).
Page 1

Add the following clause after the Scope and renumber all subsequent clauses accordingly:

2 Conformance

Biometric data interchange format conformance tests that claim conformance to this part of ISO/IEC

19794 shall satisfy the normative requirements of the methodology for those levels of test they are

claiming to perform, as described in Clauses A.1, A.2 and A.3. Any conformance tests shall use the

assertion types defined in Clause A.2 with the specific assertion details given in this and the relevant

subsequent parts of this standard.

Implementations of subsequent parts of ISO/IEC 19794 tested according to the methodology specified in

this part of ISO/IEC 19794 shall be able to claim conformance only to those requirements specified in

ISO/IEC 19794 that are tested by the test methods established by this methodology.

Pages 1 to 8, Terms and definitions

Add the following terms and definitions alphabetically , renumbering accordingly:

4.x
assertion

specification for testing a conformance requirement in an implementation under test expressed in a formal

assertion definition language
4.x
assertion test

specification of software or procedural methods that generate the test outcomes used for assessment of

conformance to an assertion

NOTE This is adapted from the definition of “assertion test” in ISO/IEC 13210:1999.

4.x
attestation

issue of a statement, based on a decision that fulfillment of specified requirements has been

demonstrated
NOTE This is adapted from the definition of “attestation” in ISO/IEC 17000:2004.
© ISO/IEC 2013 – All rights reserved
---------------------- Page: 5 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
4.x
certification
third-party attestation related to products
[ISO/IEC 17000:2004]
4.x
conformance
conformity

fulfillment by a product, process, or service of all relevant specified conformance requirements

NOTE For all practical purposes, data records are considered to be a type of a “product”, i.e., provisions of this

standard that are applicable to “products” apply to data records.
4.x
conformance requirement

requirement stated in a data format specification and defined in a finite, measurable, and unambiguous

manner

NOTE This is adapted from the definition of “conformance requirement” in ISO/IEC 13210:1999.

4.x
conformance test
specified technical procedure of conformance testing
4.x
conformance testing
testing

determination of one or more characteristics of an object of conformity assessment, according to a

procedure
[ISO/IEC 17000:2004]
4.x
conformance testing laboratory
organization that carries out conformance testing.

EXAMPLE This may be the creator of the IUT, the user of the IUT, or an unbiased third party.

4.x
conformance test specification
test specification

provisions of ISO/IEC 19794 biometric data interchange format standard that is concerned with test

methods, sometimes supplemented with other provisions related to testing, such as sampling, use of

statistical methods, and sequence of tests

NOTE This is adapted from the definition of “testing standard” in ISO/IEC GUIDE 2:2004.

4.x
conformance testing suite
CTS
test software used to automate certain types of conformance testing
4.x
conformity assessment

demonstration that specified requirements relating to a product, process, system, person or body are

fulfilled
[ISO/IEC 17000:2004]
© ISO/IEC 2013 – All rights reserved 3
---------------------- Page: 6 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
4.x
data format specification

provisions of ISO/IEC 19794 biometric data interchange format standard containing the specification that

is the subject of the conformance testing
4.x
declaration
declaration of conformity
first-party attestation
[ISO/IEC 17000:2004]
4.x
Implementation conformance statement
ICS

statement by the supplier of an implementation under test that indicates which mandatory and optional

components of the data format specification are supported by the implementation
4.x
implementation under test
IUT
that which implements the data format specification being tested

NOTE Depending on the conformance requirements of the data format specification, this may simply be a set of

biometric data interchange records or it may be a computer algorithm in the form that creates the BDIR and/or uses

the data contained in the BDIR.
4.x
input biometric data record
IBDR

data package containing a less processed form of biometric data which is suitable for use in the creation

of a BDIR

NOTE In some cases, this may be an image, but it may also be raw sensor output such as a time series of data

points from a digitization tablet.
4.x
level 1 testing

conformance testing methodology that checks field by field and byte by byte conformance with the

specification of the BDIR as specified in the data format specification, both in terms of fields included and

the ranges of the values in those fields

NOTE This type of testing tests syntactic requirements of the data format specification.

4.x
level 2 testing

conformance testing methodology that tests the internal consistency of the BDIR under test, relating

values from one part or field of the BDIR to values from other parts or fields of the BDIR

NOTE This type of testing tests syntactic requirements of the data format specification.

4.x
level 3 testing

conformance testing methodology that tests that a BDIR produced by an IUT is a faithful representation of

the IBDR subject to the constraints of the parameters in the metadata records

NOTE This type of testing tests semantic requirements of the data format specification.

© ISO/IEC 2013 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
4.x
metadata record

data record containing any specific parameters related to the data itself, particularly required by an IUT to

transform an IBDR into a BDIR

EXAMPLE Type of image (basic, full frontal, token frontal or other) and the level of compression for a face image

BDIR; the presence of core, delta, or ridgecounts in the extended area for finger minutiae BDIR; the size of each

pattern in a finger pattern BDIR.
4.x
procedure
specified way to carry out an activity or a process
[ISO 9000:2005]
4.x
requirement
provision that conveys criteria to be fulfilled
[ISO/IEC GUIDE 2:2004]
4.x
test method
specified technical procedure for performing a test

NOTE This is adapted from the definition of “test method” in ISO/IEC GUIDE 2:1996. More recent edition of

ISO/IEC GUIDE 2 or ISO/IEC 17000:2004 no longer contain this definition
4.x
test report

document that presents test results and other information relevant to the execution of the test methods

against an Implementation Under Test

NOTE This is adapted from the definition of “test report” in ISO/IEC 13210:1999 and ISO/IEC GUIDE 2:1996.

4.x
Type A conformance claim

conformance claim that an IUT is a conformant BDIR, or can create conformant BDIRs from appropriate

IBDRs
4.x
Type B conformance claim

conformance claim that an IUT can read conformant BDIRs, interpret them correctly, and perform its

desired function upon them
Page 8, Abbrevaited terms
Add the following abbreviated terms alphabetically:
CTS Conformance Testing Suite
ICS Implementation Conformance Statement
IUT Implementation Under Test
IBDR Input Biometric Data Record
© ISO/IEC 2013 – All rights reserved 5
---------------------- Page: 8 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
Page 14, 8.1 (now renumbered as 9.1)
In the last sentence, change "Annex A" to "Annex B".
Page 25, Annex A
Rename Annex A as Annex B.
Page 24
After Table 7, insert the following annex:
© ISO/IEC 2013 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
Annex A
(normative)
Conformance testing methodology
A.1 Conformance testing framework
A.1.1 Limitations

While conformance of individual elements of each data interchange record to relevant requirements can be

determined, no conformance test of a given system generating or using biometric data interchange records

can be complete or perfect. Ultimately, it is only possible to prove that an IUT is non-conformant. The goal of

conformance testing is therefore to capture enough of the requirements of the data format specification and

test them under enough conditions, that any IUT that passes the conformance test is likely to be conformant.

Two problems with a data format specification that may only become apparent during conformance testing are

that some areas may be undefined (so that the specification of these areas is left to each vendor) or ill-defined

(so that there is a contradiction between parts of the data format specification or an easy misinterpretation

caused by the wording of the data format specification). The latter problem may be resolved by an

amendment to the standard, but the former problem may be difficult to resolve. An obvious example is the use

of proprietary extended data blocks within a BDIR. There may be good reasons to allow such proprietary data,

but very little conformance testing is possible while the data remains proprietary. Also, if the data format

specification includes a requirement to interpret the BDIR or use it for biometric comparison, then it is difficult

to be sure of the effect of a proprietary data block produced by one IUT when another IUT is attempting to

interpret it.
A.1.2 Managing data records

Note that since CBEFF conformance testing is out of scope for this standard, it is generally assumed that the

BDIRs will have been removed from any CBEFF data structures prior to beginning the conformance test.

Typically, for Type A testing as described in Clause A.1.3, either the IUT will provide BDIRs without a CBEFF

encapsulator or the CTS will remove them from such an encapsulator if one exists. Regardless of the method

used, the test shall provide a means of passing the CBEFF format type corresponding to the IBDRs in the IUT

or produced by the IUT to the CTS. This may be as simple as the supplier of the IUT sending a written

instruction to the testing laboratory that all BDIRs produced by this IUT would have a particular format type, or

it may involve the IUT passing a special parameter or using a specific CBEFF patron format that is not part of

its normal function outside the test. The reason this is required is that several parts of ISO/IEC 19794 have

different format types that determine whether or not certain optional data is present. Thus format type is an

extra field that shall always be present together with a BDIR when conformance testing using that BDIR

occurs.
A.1.3 Conformance testing types

Generally, the goal of biometric data interchange format conformance testing is to assure the users of

conformant biometric products that a BDIR produced by any conformant product can be interpreted and used

correctly by any other conformant product. There are thus two types of fundamental conformance claims.

Type A is the ability to produce conformant BDIRs and Type B is the ability to use conformant BDIRs.

Different IUTs may have different purposes for which they use a conformant BDIR, and thus Type B testing is

more complex than Type A testing because it has to account for all of these purposes. ISO/IEC 19794 is

focused on Type A testing exclusively. When biometric data interchange records themselves are tested in the

absence of any software or hardware that produced them or uses them, this is treated as Type A testing.

© ISO/IEC 2013 – All rights reserved 7
---------------------- Page: 10 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)
A.1.4 Conformance testing levels
A.1.4.1 Hierarchy of Conformance Tests

A first step towards the goal of demonstrating conformance is ensuring that all of the specified fields and data

structures in the BDIR are correct and self-consistent. This does not validate the fidelity of the information

contained in the BDIR, however, since that depends on the relationship between the original IBDR and the

BDIR. This leads to a natural hierarchy of conformance testing levels.

The conformance testing hierarchy presented in this standard has three levels. Generally they progress from

least complex and expensive to test to most complex and expensive to test. They also progress from less

useful in predicting the performance of real world systems using conformant products, to more useful,

although even Level 1 conformance testing represents a significant step towards that goal. The types of

assertions for Level 1 and Level 2 testing for all parts of ISO/IEC 19794 are similar and so a list of assertion

operators and operands that should be used to define assertions is given in Clause A.2 of this part of

ISO/IEC 19794. The details of all the Level 1 and Level 2 assertions for each data format specification are

given in the appropriate subsequent parts of ISO/IEC 19794. For the more complex Level 3 testing, where the

actual fidelity of the information in the BDIR is compared to that in the IBDR, the subsequent parts of

ISO/IEC 19794 provide, as far as it is possible, guidance on how to carry out Level 3 testing for their specific

data interchange formats. A given conformance test may therefore involve conformance testing at different

levels.

It will ultimately be up to application profiles or to individual users of ISO/IEC 19794 to determine which level

of conformance testing will be required for a specific application, as well as any requirements on performance

or interoperability. This will be dependent on time, cost, and importance of biometric performance, implications

of non-interoperability and the current state of the published versions of the various parts of ISO/IEC 19794.

A.1.4.2 Level 1 – Data format conformance

In Level 1 testing, a set of BDIRs shall be checked for field-by-field and byte-by-byte conformance with the

specification of the data format specification, both in terms of fields included and the ranges of the values in

those fields. The specific assertions tested for each data format specification shall be those described in the

appropriate part of ISO/IEC 19794.

The advantage of this testing is that it does not require an IUT to be a computer algorithm or a set of hardware

and software. It can simply be a set of BDIRs. Thus, any hardware or software components of the

implementation being tested do not have to come into the possession of the testing lab, only BDIRs created

with those components.

An IUT may have the capability to produce multiple BDIRs, depending on the requirements of the application

in which it is used. Some of these BDIRs may be conformant and others may not, and so it is important to

specify which types were tested and how many of each type. In an ideal world every possible combination of

parameters for a particular biometric data interchange format would be tested, but this is not realistic given the

resources that would be required for such testing. Provided a test reports the presence or absence of optional

fields and the values for variable structural fields it is possible for a user of the data format specification to

determine if the particular variant of the standard tested is appropriate for their needs. The user may also

require conformance test results for a specific type of BDIR. An obvious example would be a two finger BDIR

or two iris BDIR, since many applications require enrollment of more than one biometric characteristic in order

to allow for a back-up if one of them becomes damaged or temporarily unusable. Some IUTs might be

conformant with a single-representation BDIR, or even with multiple representations of a single finger or iris,

but might fail conformance testing when the BDIR contains more than one finger or iris.

A.1.4.3 Level 2 – Internal consistency checking

In Level 2 testing, a set of BDIRs shall be checked to determine if they are internally consistent. The specific

assertions tested for each data format specification shall be those described in the appropriate part of

ISO/IEC 19794.
© ISO/IEC 2013 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)

The nature of Level 2 testing is that it relates values from one part of the BDIR to values from other parts of

the BDIR. This may be due to explicit requirements in the data format specification, such as a requirement

that the record length actually does indicate the number of bytes in the BDIR. It may also be implicit in the

standard, such as determining that the coordinates of a particular feature (such as eye positions in a face

image record or minutiae positions in a finger minutiae record) actually fall within the specified size of the

image.

In some cases, test assertions for Level 2 and higher conformance testing will have to make specific

assumptions about interpreting the data format specification requirements. In those cases, testing

methodology as set forth in ISO/IEC 19794 shall be considered normative in its interpretation of the data

format specification requirements, and any other interpretations shall be considered non-conformant to the

data format specification.

Once again the advantage of this testing is that it does not require an IUT to be a computer algorithm or a set

of hardware and software. It can simply be a set of BDIRs. Then the hardware or software of the IUT does not

have to be part of the test, only BDIRs created with that implementation. The disadvantage is that there are a

limited number of BDIRs and it is quite possible that some of the internal consistency checks will never be

tested because they are not relevant for the set of BDIRs in the IUT. The solution is to test a larger number of

BDIRs that represent multiple different structural variants of the biometric data interchange format under test.

This is why it is so vital to report on the structure of each BDIR variant in the conformance test.

Since Level 1 and Level 2 conformance testing are both required in order to properly test that the structure of

a BDIR is conformant to a data format specification, and since the execution of Level 1 and Level 2 tests will

frequently be intermingled, a conformance test should always include all relevant Level 1 and Level 2 test

assertions.
A.1.4.4 Level 3 – Content checking

Level 3 conformance testing is defined as a conformance testing methodology that tests that a BDIR produced

by an IUT is a faithful representation of the IBDR subject to the constraints of the parameters in the metadata

records. Effectively this is intended to test that the BDIRs produced by an IUT are faithful representations of

the original biometric data and that they satisfy those requirements of the data format specification that are not

simply a matter of syntax and format. In some cases the requirements of the data format specification may

specify biometric data capture conditions. An example would be the use of a fingerprint sensor of a particular

resolution or having a particular certification by an external body to capture fingerprints and generate

fingerprint image records that are noted in the BDIR as having a particular image acquisition level. The only

way to test that such a requirement has been correctly implemented by the IUT is to require that in Level 3

testing, an IUT shall be a combination of computer hardware and/or software that is used in the testing

laboratory. If the IUT is software only, then a set of IBDRs and corresponding metadata records shall be

provided and the IUT shall produce a set of corresponding BDIRs. This, however, only tests the ability of the

IUT to parse the metadata records and the IBDRs and insert the appropriate information in correctly formatted

BDIRs. For some requirements it is essential that the entire process from data capture to BDIR production be

included in the test, and in those cases the IUT will have to be a complete set of hardware and software.

Finally, there are some requirements that cannot be quantitatively tested without significant special effort or

extra equipment and for which the conformance test specification may simply define that no Level 3 test is

possible. An example would be the pose angles listed in a face image data record. Without an external three

dimensional reference frame for the head containing the face that is represented in the data record, there is

no absolute mechanism to verify that these pose angles are correct.

This leads to the following methodology for handling Level 3 conformance tests. As indicated in Clause A.3, all

of the requirements of the data format specification shall be listed in a conformance requirements table that is

in the same form as an implementation conformance statement. This will help the supplier of the IUT to clearly

identify which requirements of the data format specification are supported by the implementation. Certain

columns in this table indicate whether each requirement is a Level 1, Level 2 or Level 3 requirement and for

each Level 3 requirement, indicate whether it can be tested using a software only solution with a database of

IBDRs and metadata records, whether it requires a complete hardware and software IUT, or whether it can’t

be tested at all without special effort. Where they are available, the specific test methodologies to be used for

Level 3 testing involving IUTs composed of both hardware and software are found within specific clauses, as

referenced in the table, in each of the subsequent parts of ISO/IEC 19794. A general methodology to test

© ISO/IEC 2013 – All rights reserved 9
---------------------- Page: 12 ----------------------
ISO/IEC 19794-1:2011/Amd.1:2013(E)

some Level 3 requirements using a software only solution and a database of IBDRs and metadata records is

described in this clause, but even it will require specific clauses in the subsequent parts of ISO/IEC 19794 on

how to use this methodology for specific requirements.

The basic structure of a software only Level 3 conformance test is that a set of IBDRs and corresponding

metadata records shall be provided and the IUT shall produce a set of corresponding BDIRs. The information

in the BDIRs shall then be compared to the information in the IBDRs to determine if the IUT has faithfully

reproduced that in
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.