ISO/IEC TR 24722:2015
(Main)Information technology — Biometrics — Multimodal and other multibiometric fusion
Information technology — Biometrics — Multimodal and other multibiometric fusion
ISO/IEC TR 24722:2015 contains descriptions of and analyses of current practices on multimodal and other multibiometric fusion, including (as appropriate) references to more detailed descriptions. ISO/IEC TR 24722:2015 contains descriptions and explanations of high-level multibiometric concepts to aid in the explanation of multibiometric fusion approaches including multi-characteristic-type, multiinstance, multisensorial, multialgorithmic, decision-level and score-level logic.
Technologies de l'information — Biométrie — Fusion multimodale et autre fusion multibiométrique
General Information
Relations
Standards Content (Sample)
TECHNICAL ISO/IEC TR
REPORT 24722
Second edition
2015-12-15
Information technology —
Biometrics — Multimodal and other
multibiometric fusion
Technologies de l’information — Biométrie — Fusion multimodale et
autre fusion multibiométrique
Reference number
ISO/IEC TR 24722:2015(E)
©
ISO/IEC 2015
---------------------- Page: 1 ----------------------
ISO/IEC TR 24722:2015(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2015, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2015 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC TR 24722:2015(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Terms and definitions . 1
3 Overview of multimodal and other multibiometric systems . 3
3.1 General . 3
3.2 Simultaneous and sequential presentation . 5
3.2.1 General multibiometric system model. 5
3.2.2 Simultaneous presentation . 5
3.2.3 Sequential presentation. 6
3.3 Correlation . 6
4 Levels of combination . 7
4.1 Overview . 7
4.2 Decision-level fusion . 9
4.2.1 Simple decision-level fusion . 9
4.2.2 Advanced decision-level fusion . 9
4.3 Score-level fusion .11
4.3.1 Overview .11
4.3.2 Score normalization .11
4.3.3 Score fusion methods.14
4.4 Feature-level fusion .17
5 Characterisation data for multibiometric systems .17
5.1 Overview .17
5.2 Use of characterisation data in normalisation and fusion .17
Bibliography .19
© ISO/IEC 2015 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC TR 24722:2015(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/IEC JTC 1, Information technology, Subcommittee
SC 37, Biometrics.
This second edition cancels and replaces the first edition (ISO/IEC/TR 24722:2007), which has been
technically revised with the following changes:
— the original Clause 2 (Terminology issues) and Clause 7 (Scope and options for standardisation) are
removed in this edition;
— Clause 2 (Terms and definitions) is aligned with ISO/IEC 2382-37;
— the current Clause 3, Clause 4, and Clause 5 have been technically revised in terminology, the state
of arts updates, and other aspects. Such modifications have also been reflected in the bibliography.
iv © ISO/IEC 2015 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC TR 24722:2015(E)
Introduction
Some applications of biometrics require a level of technical performance that is difficult to obtain with
a single biometric measure. Such applications include prevention of multiple applications for national
identity cards and security checks for air travel. In addition, provision is needed for people who are
unable to give a reliable biometric sample for some biometric characteristic types.
Use of multiple biometric measurements from substantially independent biometric sensors, algorithms,
or characteristic types typically gives improved technical performance and reduces risk. This
includes an improved level of performance where not all biometric measurements are available such
that decisions can be made from any number of biometric measurements within an overall policy on
accept/reject thresholds.
Of the various forms of multibiometric systems, the potential for multimodal biometric systems, each
[22][45]
using an independent measure, has been discussed in the technical literature since at least 1974.
Advanced methods for combining measures at the score level have been discussed in Reference [15]
and Reference [16]. At the current level of understanding, combining results at the score level typically
requires knowledge of both genuine and impostor distributions. All of these measures are highly
application dependent and generally unknown in any real system.
Research on the methods not requiring previous knowledge of the score distributions is continuing and
research on fusion at both the image and feature levels is still progressing.
Given the current state of research into those questions and the highly application-dependent and
generally unavailable data required for proper fusion at the score level, work on multibiometric
fusion can, in the meantime, be considered mature. By intention, this Technical Report is not issued
as an International Standard, in order not to force industrial solutions to conform to the methodology
described herein. However, this Technical Report revision provides a mature technical description for
developments of multibiometric systems. It will also provide a reference on multibiometric fusion for
developers of other biometric standards and implementers.
© ISO/IEC 2015 – All rights reserved v
---------------------- Page: 5 ----------------------
TECHNICAL REPORT ISO/IEC TR 24722:2015(E)
Information technology — Biometrics — Multimodal and
other multibiometric fusion
1 Scope
This Technical Report contains descriptions of and analyses of current practices on multimodal and
other multibiometric fusion, including (as appropriate) references to more detailed descriptions.
This Technical Report contains descriptions and explanations of high-level multibiometric concepts
to aid in the explanation of multibiometric fusion approaches including multi-characteristic-type,
multiinstance, multisensorial, multialgorithmic, decision-level and score-level logic.
2 Terms and definitions
The following two categories of terms are defined here:
— terms that are specific to multimodal and multibiometric systems;
— terms that are not specific to multimodal and multibiometric systems, but are required to define the
terms in the first category and not defined in the latest revision of ISO/IEC 2382-37.
For definitions of other terms in the subject field of biometrics, refer to ISO/IEC 2382-37. For the
purposes of this document, the terms and definitions given in ISO/IEC 2382-37 and the following apply.
2.1
biometric data source
information channel (e.g. sensors, characteristic types, algorithms, instances or presentations) that
is the origin of data (e.g. captured biometric sample, extracted features, comparison score, rank or
decision) treated in fusion algorithms
2.2
biometric process
automated process using one or more biometric characteristics of a single individual for the purpose of
enrolment, verification, or identification
2.3
biometric fusion
combination of information from multiple sources, i.e., sensors, characteristic types, algorithms,
instances or presentations
2.4
cascaded system
system where pass/fail thresholds of biometric samples are used to determine if additional biometric
samples are required to reach an overall system decision
2.5
layered system
system where individual biometric scores are used to determine the pass/fail thresholds of other
biometric data processing
2.6
multialgorithmic
using multiple algorithms for processing the same biometric sample
© ISO/IEC 2015 – All rights reserved 1
---------------------- Page: 6 ----------------------
ISO/IEC TR 24722:2015(E)
2.7
multibiometric
uses multiple biometrics that can be combined at image, feature, score and/or decision level
Note 1 to entry: Multibiometric has five distinct subcategories: multi-characteristic-type (2.10), multiinstance
(2.11), multisensorial (2.13), multialgorithmic (2.6) and multipresentation (2.12).
2.8
multibiometric process
biometric process (2.2) involving the use of biometric fusion (2.3)
2.9
multibiometrics
automated recognition of individuals based on their biological or behavioral characteristics and
involving the use of biometric fusion (2.3)
2.10
multi-characteristic-type
multi-type
using information from multiple types of biometric characteristics
EXAMPLE Biometric characteristics types include: face, voice, finger, iris, retina, hand geometry,
signature/sign, keystroke, lip movement, gait, vein, DNA, ear, foot, scent, etc.
2.11
multiinstance
using multiple biometric instances within one biometric characteristic type
EXAMPLE Iris (left) + Iris (right), Fingerprint (left index) + Fingerprint (right index).
2.12
multipresentation
using either multiple presentation samples of one instance of a biometric characteristic or a single
presentation that results in the capture of multiple samples
EXAMPLE Several frames from video camera capture of a face image (possibly but not necessarily
consecutive).
Note 1 to entry: Multipresentation biometrics is considered a form of multibiometrics (2.9), if fusion techniques
are employed. Many fusion and normalisation techniques are appropriate to the integration of information from
multiple presentations of the same biometric instance.
2.13
multisensorial
using multiple sensors for capturing samples of one biometric instance
EXAMPLE For face: infrared spectrum, visible spectrum, 2-D image, and 3-D image; for fingerprint: optical,
electrostatic, and acoustic sensors.
2.14
sequential presentation
capturing biometric samples in separate capture events to be used for biometric fusion (2.3)
2.15
simultaneous presentation
capturing biometrics samples in a single capture event to be used for biometric fusion (2.3)
2 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC TR 24722:2015(E)
3 Overview of multimodal and other multibiometric systems
3.1 General
In general, the use of the terms multimodal or multibiometric indicates the presence and use of more
than one characteristic type, sensor, instance, and/or algorithm in some form of combined use for
making a specific biometric identification or verification decision. The methods of combining multiple
samples, comparison scores or comparison decisions can be very simple or mathematically complex.
For the purpose of this Technical Report, any method of combination will be considered a form of
“fusion”. Combination techniques will be covered in Clause 4.
Multimodal biometrics were first proposed, implemented and tested in the 1970s. Combining measures
was seen as a necessary future requirement for biometric systems. It was widely thought that
combining multiple measures could increase either security by decreasing the false acceptance rate
or user convenience by decreasing the false rejection rate. These systems did not seem to advance into
practical applications.
The use of fusion and related methods has been a key tool in the successful implementation of large-
scale automated fingerprint identification systems (AFISs), starting in the 1980s. Until recently,
multiple characteristic types have not been used in AFIS; however, most methods of fusion discussed
elsewhere in this Technical Report have been successfully implemented using fingerprints alone. Some
of the ways that fusion has been implemented in AFISs include the following:
— image (also known as sample) fusion in creating a single “rolled” image from a series of plain
impressions on a livescan device;
— template fusion in the use of multiple feature extraction algorithms on each fingerprint image;
— multiinstance fusion in the use of fingerprints from all ten fingers;
— multipresentation fusion in the use of rolled and slap (plain) fingerprints;
— algorithm fusion for the purpose of efficiency (cost, computational complexity, and throughput
rate); generally, comparators are used as a series of filters in order of increasing computational
complexity. These are generally implemented as a mix of decision and score-level fusion;
— algorithm fusion for the purpose of accuracy (decreasing false accept rate and/or false reject
rate, lessening sensitivity to poor-quality data); comparators are used in parallel, with fusion of
resulting scores.
The use of fusion has made AFIS possible because of fusion’s potential in improving both accuracy
and efficiency.
Most work to date on multibiometrics has focused only on improving false acceptance and false
rejection error rates. Some research work considers the use of multibiometrics to flexibly improve
[64]
usability, security or accuracy. Further, multibiometrics also aims at decreasing the overall failure-
to-enrol rate (FTE) especially in biometric systems where user cooperation is not expected (e.g. video
surveillance systems). Multibiometrics is an effort to produce a biometric decision even if only a subset
[66]
of the expected biometric characteristics were captured.
To further the understanding of the distinction among the multibiometric categories, Table 1 illustrates
the basic distinctions among categories of multibiometric implementation. The key aspect of the
category that makes it multi-“something” is shown in boldface.
© ISO/IEC 2015 – All rights reserved 3
---------------------- Page: 8 ----------------------
ISO/IEC TR 24722:2015(E)
Table 1 — Multibiometric categories illustrated by the simplest case of using 2 of something
Characteristic
Category Algorithm Instance Sensor
type
Multi-characteris- 2 2 2 2
b
tic-type (always) (always) (always) (usually)
1 2 1 1
Multialgorithmic
(always) (always) (always) (always)
1 1 2 1
Multiinstance
c
(always) (always) (always) (usually)
1 1 1 2
Multisensorial
a
(always) (usually) (always, and same instance) (always)
Multipresentation 1 1 1 1
a
It is possible that two samples from separate sensors could be processed by separate “feature extraction” algorithms,
and then through a common comparison algorithm, making this “1.5 algorithms”, or two completely different algorithms.
b
Exception: a multi-characteristic-type system with a single sensor used to capture two different characteristic types.
For example, a high-resolution image used to extract face and iris or face and skin texture.
c
Exception may be the use of two individual sensors to each capture one instance, for example, possibly a two-finger
fingerprint sensor.
Multi-characteristic-type biometric systems take input from single or multiple sensors that capture
two or more different types of biometric characteristics. For example, a single system combining
face and iris information for biometric recognition would be considered a “multi-characteristic-type”
system regardless of whether face and iris images were captured by different imaging devices or the
same device. It is not required that the various measures be mathematically combined in anyway. For
example, a system with fingerprint and voice recognition would be considered “multi-characteristic-
type” even if the “OR” rule was being applied, allowing users to be verified using either of the
characteristic types.
Multialgorithmic biometric systems receive a single sample from a single sensor and process that
sample with two or more algorithms. This technique could be applied to any characteristic type.
Maximum benefit (theoretically) would be derived from algorithms that are based on distinctly
different and independent principles such as either features they extract from the biometric sample
(e.g. finger minutiae versus finger pattern) or approaches to comparison (e.g. different algorithms
comparing minutiae).
Multiinstance biometric systems use one (or possibly multiple) sensor(s) to capture samples of two
or more different instances of the same biometric characteristic. For example, systems capturing
images from multiple fingers are considered to be multiinstance rather than multi-characteristic-type.
However, systems capturing, for example, sequential frames of facial or iris images are considered to be
multipresentation rather than multiinstance.
Multisensorial biometric systems sample the same instance of a biometric characteristic with two or
more distinctly different sensors. Processing of the multiple samples can be done with one algorithm,
or some combination of multiple algorithms. For example, a face recognition application could use both
a visible light camera and an infrared camera coupled with a specific frequency (or several frequencies)
of infrared illumination.
For a specific application in an operational environment, there are numerous system design
considerations, and trade-offs that should be made, among factors such as improved performance
(e.g. identification or verification accuracy, system speed and throughput, robustness, and resource
requirements), acceptability, circumvention, ease of use, operational cost, environmental flexibility,
[40]
and population flexibility.
Especially for a large-scale human identification system, there are additional system design
considerations such as operation and maintenance, reliability, system acquisition cost, life cycle cost,
and planned system response to identified susceptible means of attack, all of which will affect the
[40]
overall deployability of the system.
4 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC TR 24722:2015(E)
3.2 Simultaneous and sequential presentation
3.2.1 General multibiometric system model
A general multibiometric system model is shown in Figure 1. For explanatory purposes, this model
uses three biometric samples (P1, P2, P3) from three unique biometric characteristic types, except for
where specified differently. At the topmost level, a subject presents their biometric characteristic(s) to
the system. Dependent upon the system design, there are two methods of presenting characteristics for
acquisition by the system: simultaneous and sequential.
NOTE The presentation (simultaneous or sequential) method induce or general different fusion process.
The purpose of including this information is to illustrate considerations that can influence multibiometric
system design.
Figure 1 — Multibiometric system model
3.2.2 Simultaneous presentation
Simultaneous presentation (with successful capture) provides biometric sample(s) from multiple
characteristic types in a single event (e.g. a face and iris taken from the same camera). System designs
that utilize simultaneous acquisition would tend toward high throughput applications at the expense
of possible added complexity (to synchronize sample collection) or difficulty of use (dual sensor
interaction, user multi-tasking).
© ISO/IEC 2015 – All rights reserved 5
---------------------- Page: 10 ----------------------
ISO/IEC TR 24722:2015(E)
3.2.3 Sequential presentation
Sequential capture acquires biometric sample(s) from one or multiple characteristic types in separate
events. Sequential capture may be utilized in three concepts discussed in the literature. The first is
multiinstance, which is the use of two or more instances within one characteristic type for a subject,
i.e. Fingerprint (left index) + Fingerprint (right index). In this example, one single digit fingerprint
reader is used twice in sequence. The second concept is multi-characteristic-type, which is the use of
multiple different biometric characteristic types captured from one or more sensors for a subject, i.e.
Hand + Face in sequence. The third concept is multisensorial, which is the use of two or more distinct
sensors for capturing the same biometric feature(s) for a subject, but not at the same time. To avoid
confusion with multi-characteristic-type, which may also capture biometric feature(s) from two or
more distinct sensors, multisensorial can be clarified as “uni-characteristic-type multisensorial”.
Examples for face recognition are infrared spectrum, visible spectrum, 2-D image, and 3-D image; for
fingerprint recognition: optical, electrostatic and acoustic sensors.
3.3 Correlation
In multimodal biometric systems, the information being fused may be correlated at several different
[56]
levels as illustrated in the following examples.
— Correlation between characteristic types: This refers to biometric samples that are physically
related, such as the speech and lip movement of a user.
— Correlation due to identical biometric samples: This is the case in multialgorithmic systems where
the same biometric sample (e.g. a fingerprint image) or sub-sets of the biometric sample (e.g. voice,
where an entire sample may be used by one algorithm and part of the sample by another) is subjected
to different feature extraction and comparison algorithms (e.g. a minutiae-based comparator and a
texture-based comparator).
— Correlation between feature values: A subset of feature values constituting the feature vectors
of different characteristic types may be correlated. For example, the area of a user’s palm (hand
geometry) may be correlated with the width of the face.
— Correlation among instances due to common operating procedures (e.g. common capture device
and operator training).
— Correlation among instances due to subject behaviour (e.g. coloured contact lenses on both eyes).
However, in order to determine the extent of correlation, it is necessary to examine the comparison
scores (or the ACCEPT/REJECT decision) pertaining to the comparators involved in the fusion scheme.
In the multiple classifier system literature, it has been demonstrated that fusing uncorrelated classifiers
[56]
leads to a significant improvement in comparison performance.
For two classifiers of reasonable accuracy involved in a fusion scheme, score outputs from inputs that
come from the same subject may, but need not, be correlated. Therefore it is more appropriate to
consider the correlation of classifier errors as described by Reference [20]. The correlation ρ is given
n
c
by Formula (1):
f
nN
c
ρ = (1)
n
t f f
c
NN−−Nn+ N
c c c
where
n is the number of classifiers under test;
N is the total number of sequences;
6 © ISO/IEC 2015 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC TR 24722:2015(E)
f
N is the number of sequences where all classifiers have an incorrect output at threshold C;
c
t
N
is the number of sequences where all classifiers have a correct output at a threshold C.
c
NOTE This expression is relevant for computing the correlation of errors at the decision level.
4 Levels of combination
4.1 Overview
As a basis for the definition of levels of combination in multibiometric systems, we first introduce
the single-biometric process and its building blocks, using the example of an authentication system.
Figure 2 shows the block diagram of a single-biometric process.
Figure 2 — Single biometric process (generic)
A biometric sample captured by a biometric sensor (e.g. a fingerprint image) is fed into the Feature
Extraction module. Using signal processing methods, the feature extraction module converts a sample
into Features (e.g. fingerprint minutiae), which form a representation apt for comparison. Usually,
multiple features are collected into a feature vector. The Comparison module takes the feature vector
as input and compares it to a Biometric Reference. The result is a comparison Score, which is used by
the Decision module to decide (e.g. by applying a threshold) whether the presented sample matches
with the stored template. The outcome of this decision is a binary match or non-match.
Generalizing the above process to multiple biometrics, there are several levels at which fusion can take
place.
These include consolidating information at the (a) decision level, (b) comparison score level, (c) feature
level, and (d) sample level. Note that fusion at levels (a) and (b) occur after the comparison module is
invoked, while levels (c), and (d) occur before the comparator. Although integration is possible at these
different levels, fusion at the feature set level, the comparison score level and the decision level are
the most commonly used. Figure 3 illustrates the diffe
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.