ISO/IEC 15938-16:2021

Information technology — Multimedia content description interface — Part 16: Conformance and reference software for compact descriptors for video analysis

This document specifies the assessment of conformance to ISO/IEC 15938-15 as well as the reference software.

Technologies de l'information — Interface de description du contenu multimédia — Partie 16: Conformité et logiciels de référence pour les descripteurs compacts pour l'analyse vidéo

General Information

• Status: Published
• Publication Date: 06-Jun-2021
• ICS: 35.040.40 - Coding of audio, video, multimedia and hypermedia information
• Technical Committee: ISO/IEC JTC 1/SC 29 - Coding of audio, picture, multimedia and hypermedia information
• Drafting Committee: ISO/IEC JTC 1/SC 29/WG 4 - MPEG Video coding
• Current Stage: 6060 - International Standard published
• Start Date: 07-Jun-2021
• Due Date: 15-Nov-2020
• Completion Date: 07-Jun-2021

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 15938-16
First edition
2021-06
Information technology — Multimedia
content description interface —
Part 16:
Conformance and reference software
for compact descriptors for video
analysis
Technologies de l'information — Interface de description du contenu
multimédia —
Partie 16: Conformité et logiciels de référence pour les descripteurs
compacts pour l'analyse vidéo
Reference number
ISO/IEC 15938-16:2021(E)
©
ISO/IEC 2021

---------------------- Page: 1 ----------------------
ISO/IEC 15938-16:2021(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2021 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC 15938-16:2021(E)

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms, relational operators, bitwise operators and functions .1
5 Conformance testing . 1
5.1 Conformance video dataset and reference bitstreams . 1
5.2 Conformance test conditions for strict conformance . 2
5.2.1 Test setup . 2
5.2.2 Test conditions. 2
5.2.3 Global descriptor test . 2
5.2.4 Local descriptor test . 2
5.2.5 Deep feature descriptor test with default NN . 2
5.2.6 Combined global and local descriptor test . 2
5.2.7 Combined global and deep feature descriptor test with default NN. 3
5.3 Conformance test conditions using custom NN . 3
6 Reference software . 3
6.1 Reference software location . 3
6.2 Reference software licence . 3
6.3 Reference software documentation . 3
6.4 Reference software installation and compilation . 3
6.5 Reference software architecture . 4
6.5.1 General. 4
6.5.2 Extraction . 5
6.5.3 Pairwise matching . 7
6.5.4 Retrieval . 8
6.5.5 Parameters . 9
6.5.6 Implementation details for CDVA reference software .10
6.5.7 Usage .10
Annex A (informative) Deployment of the reference software using Docker containers.14
© ISO/IEC 2021 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC 15938-16:2021(E)

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members
_experts/ refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC
list of patent declarations received (see patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/
iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 15938 series can be found on the ISO website.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national
-committees.
iv © ISO/IEC 2021 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC 15938-16:2021(E)

Introduction
ISO/IEC 15938 (all parts) provides a standardized set of technologies for describing multimedia content.
It addresses a broad spectrum of multimedia applications and requirements by providing a metadata
system for describing the features of multimedia content.
The following are specified in ISO/IEC 15938 (all parts):
Description schemes (DS) describe entities or relationships pertaining to multimedia content.
Description schemes specify the structure and semantics of their components, which can be description
schemes, descriptors or datatypes.
Descriptors (D) describe features, attributes or groups of attributes of multimedia content.
Datatypes are the basic reusable datatypes employed by description schemes and descriptors.
Description definition language (DDL) defines description schemes, descriptors and datatypes by
specifying their syntax, and allows their extension.
Systems tools support delivery of descriptions, multiplexing of descriptions with multimedia content,
synchronization, file format, etc.
ISO/IEC 15938 (all parts) is subdivided into 16 published parts with further parts in development:
— Part 1: Systems: specifies the tools for preparing descriptions for efficient transport and storage,
compressing descriptions, and allowing synchronization between content and descriptions.
— Part 2: Description definition language: specifies the language for defining the series set of
description tools (DSs, Ds and datatypes) and for defining new description tools.
— Part 3: Visual: specifies the description tools pertaining to visual content.
— Part 4: Audio: specifies the description tools pertaining to audio content.
— Part 5: Multimedia description schemes: specifies the generic description tools pertaining to
multimedia including audio and visual content.
— Part 6: Reference software: provides a software implementation of the series.
— Part 7: Conformance testing: specifies the guidelines and procedures for testing conformance of
implementations of the series.
— Part 8: Extraction and use of MPEG-7 descriptions: provides guidelines and examples of the
extraction and use of descriptions.
— Part 9: Profiles and levels: provides guidelines and standard profiles.
— Part 10: Schema definition: specifies the schema using description definition language.
— Part 11: MPEG-7 profile schemas: listing of profile schemas using description definition language.
— Part 12: Query format: contains the tools of the MPEG query format (MPQF).
— Part 13: Compact descriptors for visual search: specifies an image description tool for visual
search applications.
— Part 14: Reference software, conformance and usage guidelines for compact descriptors for
visual search: provides the reference software and guidelines, specifies the conformance testing.
— Part 15: Compact descriptors for video analysis: specifies a video description tool designed to
enable efficient and interoperable video analysis applications, allowing visual content matching in
videos.
© ISO/IEC 2021 – All rights reserved v

---------------------- Page: 5 ----------------------
ISO/IEC 15938-16:2021(E)

— Part 16 (this document): Conformance and reference software for compact descriptors for
video analysis: describes conformance testing of descriptors specified in ISO/IEC 15938-15, and
provides the reference software for extracting and matching such descriptors.
The structure of this document is as follows:
— Clause 5 specifies conformance testing of descriptors.
— Clause 6 describes the reference software for extracting and matching descriptors. The deployment
of the reference software using Docker containers is described in Annex A.
vi © ISO/IEC 2021 – All rights reserved

---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 15938-16:2021(E)
Information technology — Multimedia content description
interface —
Part 16:
Conformance and reference software for compact
descriptors for video analysis
1 Scope
This document specifies the assessment of conformance to ISO/IEC 15938-15 as well as the reference
software.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 15938-14, Information technology — Multimedia content description interface — Part 14:
Reference software, conformance and usage guidelines for compact descriptors for visual search
ISO/IEC 15938-15, Information technology — Multimedia content description interface — Part 15:
Compact descriptors for video analysis
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 15938-15 apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
4 Abbreviated terms, relational operators, bitwise operators and functions
For the purposes of this document, the abbreviated terms, relational operators, bitwise operators and
functions in ISO/IEC 15938-15 apply.
5 Conformance testing
5.1 Conformance video dataset and reference bitstreams
The CDVA conformance data set, containing 3 318 items, is used for conformance testing.
The data set is available at https:// standards .iso .org/ iso -iec/ 15938/ -16/ ed -1/ en.
For deep feature descriptors, two levels of conformance are considered:
— Conformance using default neural network: This is a stricter level of conformance, which ensures
that the bitstreams match those generated with the reference software as-is.
© ISO/IEC 2021 – All rights reserved 1

---------------------- Page: 7 ----------------------
ISO/IEC 15938-16:2021(E)

— Conformance using custom neural network: This is a less strict level of conformance, that allows
using a custom neural network. The deep feature descriptors generated with implementations
using different neural networks are not interoperable.
5.2 Conformance test conditions for strict conformance
5.2.1 Test setup
— Bitstreams extracted with the reference software on Ubuntu 16.04 are provided.
— Implementation under test is used to extract bitstreams from the conformance test set, including all
three descriptor components, and varying the parameters listed in subclause 6.5.4.
— Use the reference implementation to perform pairwise matching between the bitstreams of
corresponding videos.
— Check that the resulting similarity is above the thresholds listed in the following subclauses.
5.2.2 Test conditions
Vary minLocalDiff for the local feature descriptor in the conformance tests. At least the following values
for minLocalDiff should be tested: 1; 31.
5.2.3 Global descriptor test
Perform pairwise matching of global descriptors.
An implementation is considered conforming to the specification, if at least 99% of the descriptors of
the conformance dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.2.4 Local descriptor test
Perform pairwise matching of local descriptors.
An implementation is considered conforming to the specification if at least 99% of the descriptors of the
conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.2.5 Deep feature descriptor test with default NN
Perform pairwise matching of deep feature descriptors, using the default NN.
An implementation is considered conforming to the specification if at least 99% of the descriptors of the
conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.2.6 Combined global and local descriptor test
Perform pairwise matching combining local and global descriptors.
An implementation is considered conforming to the specification if at least 99% of the descriptors of the
conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
2 © ISO/IEC 2021 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC 15938-16:2021(E)

with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.2.7 Combined global and deep feature descriptor test with default NN
Perform pairwise matching combining local and global descriptors, using the default NN.
An implementation is considered conforming to the specification if at least 99% of the descriptors of the
conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.3 Conformance test conditions using custom NN
As a prerequisite for this test, at least the conformance tests for global descriptors shall be run.
No reference data is provided for this test.
Only syntactic checking can be performed by using the reference software to parse and match
bitstreams generated with the implementation under test. This shall complete without errors.
6 Reference software
6.1 Reference software location
The reference software is available at https:// standards .iso .org/ iso -iec/ 15938/ -16/ ed -1/ en.
6.2 Reference software licence
The licence of the software is described in the file CDVA_evaluation_framework/COPYING included with
the reference software.
6.3 Reference software documentation
The documentation of the software is contained in the folder CDVA_evaluation_framework/docs included
with the reference software. An overview is provided in the file CDVA_evaluation_framework/README.
6.4 Reference software installation and compilation
The installation of the software is described in file CDVA_evaluation_framework/INSTALLING included
with the reference software.
The CDVA reference s
...

FINAL
INTERNATIONAL ISO/IEC
DRAFT
STANDARD FDIS
15938-16
ISO/IEC JTC 1/SC 29
Information technology — Multimedia
Secretariat: JISC
content description interface —
Voting begins on:
2021-03-09
Part 16:
Voting terminates on:
Conformance and reference software
2021-05-04
for compact descriptors for video
analysis
RECIPIENTS OF THIS DRAFT ARE INVITED TO
SUBMIT, WITH THEIR COMMENTS, NOTIFICATION
OF ANY RELEVANT PATENT RIGHTS OF WHICH
THEY ARE AWARE AND TO PROVIDE SUPPOR TING
DOCUMENTATION.
IN ADDITION TO THEIR EVALUATION AS
Reference number
BEING ACCEPTABLE FOR INDUSTRIAL, TECHNO-
ISO/IEC FDIS 15938-16:2021(E)
LOGICAL, COMMERCIAL AND USER PURPOSES,
DRAFT INTERNATIONAL STANDARDS MAY ON
OCCASION HAVE TO BE CONSIDERED IN THE
LIGHT OF THEIR POTENTIAL TO BECOME STAN-
DARDS TO WHICH REFERENCE MAY BE MADE IN
©
NATIONAL REGULATIONS. ISO/IEC 2021

---------------------- Page: 1 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2021 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms, relational operators, bitwise operators and functions .1
5 Conformance testing . 1
5.1 Conformance video dataset and reference bitstreams . 1
5.2 Conformance test conditions for strict conformance . 2
5.2.1 Test setup . 2
5.2.2 Test conditions. 2
5.2.3 Global descriptor test . 2
5.2.4 Local descriptor test . 2
5.2.5 Deep feature descriptor test with default NN . 2
5.2.6 Combined global and local descriptor test . 2
5.2.7 Combined global and deep feature descriptor test with default NN. 3
5.3 Conformance test conditions using custom NN . 3
6 Reference software . 3
6.1 Reference software location . 3
6.2 Reference software licence . 3
6.3 Reference software documentation . 3
6.4 Reference software installation and compilation . 3
6.5 Reference software architecture . 3
6.5.1 General. 3
6.5.2 Extraction . 4
6.5.3 Pairwise matching . 6
6.5.4 Retrieval . 7
6.5.5 Parameters . 8
6.5.6 Implementation details for CDVA reference software . 9
6.5.7 Usage . 9
Annex A (informative) Deployment of the reference software using Docker containers.13
© ISO/IEC 2021 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that
are members of ISO or IEC participate in the development of International Standards through
technical committees established by the respective organization to deal with particular fields of
technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other
international organizations, governmental and non-governmental, in liaison with ISO and IEC, also
take part in the work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members
_experts/ refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC
list of patent declarations received (see patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/
iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 15938 series can be found on the ISO website.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national
-committees.
iv © ISO/IEC 2021 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

Introduction
ISO/IEC 15938 (all parts) provides a standardized set of technologies for describing multimedia content.
It addresses a broad spectrum of multimedia applications and requirements by providing a metadata
system for describing the features of multimedia content.
The following are specified in ISO/IEC 15938 (all parts):
Description schemes (DS) describe entities or relationships pertaining to multimedia content.
Description schemes specify the structure and semantics of their components, which can be description
schemes, descriptors or datatypes.
Descriptors (D) describe features, attributes or groups of attributes of multimedia content.
Datatypes are the basic reusable datatypes employed by description schemes and descriptors.
Description definition language (DDL) defines description schemes, descriptors and datatypes by
specifying their syntax, and allows their extension.
Systems tools support delivery of descriptions, multiplexing of descriptions with multimedia content,
synchronization, file format, etc.
ISO 15938 (all parts) is subdivided into 16 published parts with further parts in development:
— Part 1: Systems: specifies the tools for preparing descriptions for efficient transport and storage,
compressing descriptions, and allowing synchronization between content and descriptions.
— Part 2: Description definition language: specifies the language for defining the series set of
description tools (DSs, Ds and datatypes) and for defining new description tools.
— Part 3: Visual: specifies the description tools pertaining to visual content.
— Part 4: Audio: specifies the description tools pertaining to audio content.
— Part 5: Multimedia description schemes: specifies the generic description tools pertaining to
multimedia including audio and visual content.
— Part 6: Reference software: provides a software implementation of the series.
— Part 7: Conformance testing: specifies the guidelines and procedures for testing conformance of
implementations of the series.
— Part 8: Extraction and use of MPEG-7 descriptions: provides guidelines and examples of the
extraction and use of descriptions.
— Part 9: Profiles and levels: provides guidelines and standard profiles.
— Part 10: Schema definition: specifies the schema using description definition language.
— Part 11: MPEG-7 profile schemas: listing of profile schemas using description definition language.
— Part 12: Query format: contains the tools of the MPEG query format (MPQF).
— Part 13: Compact descriptors for visual search: specifies an image description tool for visual
search applications.
— Part 14: Reference software, conformance and usage guidelines for compact descriptors for
visual search: provides the reference software and guidelines, specifies the conformance testing.
— Part 15: Compact descriptors for video analysis: specifies a video description tool designed to
enable efficient and interoperable video analysis applications, allowing visual content matching
in videos.
© ISO/IEC 2021 – All rights reserved v

---------------------- Page: 5 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

— Part 16 (this document): Conformance and reference software for compact descriptors for
video analysis: describes conformance testing of descriptors specified in ISO/IEC 15938-15, and
provides the reference software for extracting and matching such descriptors.
The structure of this document is as follows:
— Clause 5 specifies conformance testing of descriptors.
— Clause 6 describes the reference software for extracting and matching descriptors. The deployment
of the reference software using Docker containers is described in Annex A.
vi © ISO/IEC 2021 – All rights reserved

---------------------- Page: 6 ----------------------
FINAL DRAFT INTERNATIONAL STANDARD ISO/IEC FDIS 15938-16:2021(E)
Information technology — Multimedia content description
interface —
Part 16:
Conformance and reference software for compact
descriptors for video analysis
1 Scope
This document specifies the assessment of conformance to ISO/IEC 15938-15 as well as the reference
software.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 15938-14, Information technology — Multimedia content description interface — Part 14:
Reference software, conformance and usage guidelines for compact descriptors for visual search
ISO/IEC 15938-15, Information technology — Multimedia content description interface — Part 15:
Compact descriptors for video analysis
3 Terms and definitions
For the purposes of this document, the terms and definitions in ISO/IEC 15938-15 apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
4 Abbreviated terms, relational operators, bitwise operators and functions
For the purposes of this document, the abbreviated terms, relational operators, bitwise operators and
functions in ISO/IEC 15938-15 apply.
5 Conformance testing
5.1 Conformance video dataset and reference bitstreams
The CDVA conformance data set, containing 3 318 items, is used for conformance testing.
The data set is available at https:// standards .iso .org/ iso -iec/ 15938/ -16/ ed -1/ en.
For deep feature descriptors, two levels of conformance are considered:
— Conformance using default neural network: This is a stricter level of conformance, which ensures
that the bitstreams match those generated with the reference software as-is.
© ISO/IEC 2021 – All rights reserved 1

---------------------- Page: 7 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

— Conformance using custom neural network: This is a less strict level of conformance, that allows
using a custom neural network. The deep feature descriptors generated with implementations
using different neural networks are not interoperable.
5.2 Conformance test conditions for strict conformance
5.2.1 Test setup
— Bitstreams extracted with the reference software on Ubuntu 16.04 are provided.
— Implementation under test is used to extract bitstreams from the conformance test set, including all
three descriptor components, and varying the parameters listed in subclause 6.5.4.
— Use the reference implementation to perform pairwise matching between the bitstreams of
corresponding videos.
— Check that the resulting similarity is above the thresholds listed in the following subclauses.
5.2.2 Test conditions
Vary minLocalDiff for the local feature descriptor in the conformance tests. At least the following values
for minLocalDiff should be tested: 1; 31.
5.2.3 Global descriptor test
Perform pairwise matching of global descriptors.
An implementation is considered conforming to the specification, if at least 99% of the descriptors of
the conformance dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.2.4 Local descriptor test
Perform pairwise matching of local descriptors.
An implementation is considered conforming to the specification if at least 99% of the descriptors of the
conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.2.5 Deep feature descriptor test with default NN
Perform pairwise matching of deep feature descriptors, using the default NN.
An implementation is considered conforming to the specification if at least 99% of the descriptors of the
conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.2.6 Combined global and local descriptor test
Perform pairwise matching combining local and global descriptors.
An implementation is considered conforming to the specification if at least 99% of the descriptors of the
conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
2 © ISO/IEC 2021 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.2.7 Combined global and deep feature descriptor test with default NN
Perform pairwise matching combining local and global descriptors, using the default NN.
An implementation is considered conforming to the specification if at least 99% of the descriptors of the
conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted
with the implementation under test with the reference bitstreams for the same video. s is the matching
score for the same pair of videos when matching the reference bitstreams to itself using the reference
implementation.
5.3 Conformance test conditions using custom NN
As a prerequisite for this test, at least the conformance tests for global descriptors shall be run.
No reference data is provided for this test.
Only syntactic checking can be performed by using the reference software to parse and match
bitstreams generated with the implementation under test. This shall complete without errors.
6 Reference software
6.1 Reference software location
The reference software is available at https:// standards .iso .org/ iso -iec/ 15938/ -16/ ed -1/ en.
6.2 Reference software licence
The licence of the software is described in the file CDVA_evaluation_framework/COPYING included with
the reference software.
6.3 Reference software documentation
The documentation of the software is contained in the folder CDVA_evaluation_framework/docs included
with the reference software. An overview is provided in the file CDVA_evaluation_framework/README.
6.4 Re
...