Information technology -- Multimedia content description interface

This document specifies the assessment of conformance to ISO/IEC 15938-15 as well as the reference software.

Technologies de l'information -- Interface de description du contenu multimédia

General Information

Status
Published
Publication Date
06-Jun-2021
Current Stage
5060 - Close of voting Proof returned by Secretariat
Start Date
05-May-2021
Completion Date
04-May-2021
Ref Project

Buy Standard

Standard
ISO/IEC 15938-16:2021 - Information technology -- Multimedia content description interface
English language
14 pages
sale 15% off
Preview
sale 15% off
Preview
Draft
ISO/IEC FDIS 15938-16:Version 06-mar-2021 - Information technology -- Multimedia content description interface
English language
14 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

INTERNATIONAL ISO/IEC
STANDARD 15938-16
First edition
2021-06
Information technology — Multimedia
content description interface —
Part 16:
Conformance and reference software
for compact descriptors for video
analysis
Technologies de l'information — Interface de description du contenu
multimédia —
Partie 16: Conformité et logiciels de référence pour les descripteurs
compacts pour l'analyse vidéo
Reference number
ISO/IEC 15938-16:2021(E)
ISO/IEC 2021
---------------------- Page: 1 ----------------------
ISO/IEC 15938-16:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 15938-16:2021(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

Introduction ..................................................................................................................................................................................................................................v

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Abbreviated terms, relational operators, bitwise operators and functions ...........................................1

5 Conformance testing ......................................................................................................................................................................................... 1

5.1 Conformance video dataset and reference bitstreams ......................................................................................... 1

5.2 Conformance test conditions for strict conformance ............................................................................................ 2

5.2.1 Test setup ............................................................................................................................................................................... 2

5.2.2 Test conditions................................................................................................................................................................... 2

5.2.3 Global descriptor test .................................................................................................................................................. 2

5.2.4 Local descriptor test ..................................................................................................................................................... 2

5.2.5 Deep feature descriptor test with default NN ......................................................................................... 2

5.2.6 Combined global and local descriptor test ................................................................................................ 2

5.2.7 Combined global and deep feature descriptor test with default NN................................... 3

5.3 Conformance test conditions using custom NN ......................................................................................................... 3

6 Reference software ............................................................................................................................................................................................ 3

6.1 Reference software location ........................................................................................................................................................ 3

6.2 Reference software licence ........................................................................................................................................................... 3

6.3 Reference software documentation ...................................................................................................................................... 3

6.4 Reference software installation and compilation ..................................................................................................... 3

6.5 Reference software architecture .............................................................................................................................................. 4

6.5.1 General...................................................................................................................................................................................... 4

6.5.2 Extraction ............................................................................................................................................................................... 5

6.5.3 Pairwise matching .......................................................................................................................................................... 7

6.5.4 Retrieval .................................................................................................................................................................................. 8

6.5.5 Parameters ............................................................................................................................................................................ 9

6.5.6 Implementation details for CDVA reference software ..................................................................10

6.5.7 Usage .......................................................................................................................................................................................10

Annex A (informative) Deployment of the reference software using Docker containers...........................14

© ISO/IEC 2021 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 15938-16:2021(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are

members of ISO or IEC participate in the development of International Standards through technical

committees established by the respective organization to deal with particular fields of technical

activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international

organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the

work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members

_experts/ refdocs).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC

list of patent declarations received (see patents.iec.ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/

iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.

A list of all parts in the ISO/IEC 15938 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national

-committees.
iv © ISO/IEC 2021 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC 15938-16:2021(E)
Introduction

ISO/IEC 15938 (all parts) provides a standardized set of technologies for describing multimedia content.

It addresses a broad spectrum of multimedia applications and requirements by providing a metadata

system for describing the features of multimedia content.
The following are specified in ISO/IEC 15938 (all parts):

Description schemes (DS) describe entities or relationships pertaining to multimedia content.

Description schemes specify the structure and semantics of their components, which can be description

schemes, descriptors or datatypes.

Descriptors (D) describe features, attributes or groups of attributes of multimedia content.

Datatypes are the basic reusable datatypes employed by description schemes and descriptors.

Description definition language (DDL) defines description schemes, descriptors and datatypes by

specifying their syntax, and allows their extension.

Systems tools support delivery of descriptions, multiplexing of descriptions with multimedia content,

synchronization, file format, etc.

ISO/IEC 15938 (all parts) is subdivided into 16 published parts with further parts in development:

— Part 1: Systems: specifies the tools for preparing descriptions for efficient transport and storage,

compressing descriptions, and allowing synchronization between content and descriptions.

— Part 2: Description definition language: specifies the language for defining the series set of

description tools (DSs, Ds and datatypes) and for defining new description tools.

— Part 3: Visual: specifies the description tools pertaining to visual content.
— Part 4: Audio: specifies the description tools pertaining to audio content.

— Part 5: Multimedia description schemes: specifies the generic description tools pertaining to

multimedia including audio and visual content.
— Part 6: Reference software: provides a software implementation of the series.

— Part 7: Conformance testing: specifies the guidelines and procedures for testing conformance of

implementations of the series.

— Part 8: Extraction and use of MPEG-7 descriptions: provides guidelines and examples of the

extraction and use of descriptions.
— Part 9: Profiles and levels: provides guidelines and standard profiles.

— Part 10: Schema definition: specifies the schema using description definition language.

— Part 11: MPEG-7 profile schemas: listing of profile schemas using description definition language.

— Part 12: Query format: contains the tools of the MPEG query format (MPQF).

— Part 13: Compact descriptors for visual search: specifies an image description tool for visual

search applications.

— Part 14: Reference software, conformance and usage guidelines for compact descriptors for

visual search: provides the reference software and guidelines, specifies the conformance testing.

— Part 15: Compact descriptors for video analysis: specifies a video description tool designed to

enable efficient and interoperable video analysis applications, allowing visual content matching in

videos.
© ISO/IEC 2021 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC 15938-16:2021(E)

— Part 16 (this document): Conformance and reference software for compact descriptors for

video analysis: describes conformance testing of descriptors specified in ISO/IEC 15938-15, and

provides the reference software for extracting and matching such descriptors.
The structure of this document is as follows:
— Clause 5 specifies conformance testing of descriptors.

— Clause 6 describes the reference software for extracting and matching descriptors. The deployment

of the reference software using Docker containers is described in Annex A.
vi © ISO/IEC 2021 – All rights reserved
---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 15938-16:2021(E)
Information technology — Multimedia content description
interface —
Part 16:
Conformance and reference software for compact
descriptors for video analysis
1 Scope

This document specifies the assessment of conformance to ISO/IEC 15938-15 as well as the reference

software.
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO/IEC 15938-14, Information technology — Multimedia content description interface — Part 14:

Reference software, conformance and usage guidelines for compact descriptors for visual search

ISO/IEC 15938-15, Information technology — Multimedia content description interface — Part 15:

Compact descriptors for video analysis
3 Terms and definitions

For the purposes of this document, the terms and definitions given in ISO/IEC 15938-15 apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
4 Abbreviated terms, relational operators, bitwise operators and functions

For the purposes of this document, the abbreviated terms, relational operators, bitwise operators and

functions in ISO/IEC 15938-15 apply.
5 Conformance testing
5.1 Conformance video dataset and reference bitstreams

The CDVA conformance data set, containing 3 318 items, is used for conformance testing.

The data set is available at https:// standards .iso .org/ iso -iec/ 15938/ -16/ ed -1/ en.

For deep feature descriptors, two levels of conformance are considered:

— Conformance using default neural network: This is a stricter level of conformance, which ensures

that the bitstreams match those generated with the reference software as-is.
© ISO/IEC 2021 – All rights reserved 1
---------------------- Page: 7 ----------------------
ISO/IEC 15938-16:2021(E)

— Conformance using custom neural network: This is a less strict level of conformance, that allows

using a custom neural network. The deep feature descriptors generated with implementations

using different neural networks are not interoperable.
5.2 Conformance test conditions for strict conformance
5.2.1 Test setup
— Bitstreams extracted with the reference software on Ubuntu 16.04 are provided.

— Implementation under test is used to extract bitstreams from the conformance test set, including all

three descriptor components, and varying the parameters listed in subclause 6.5.4.

— Use the reference implementation to perform pairwise matching between the bitstreams of

corresponding videos.

— Check that the resulting similarity is above the thresholds listed in the following subclauses.

5.2.2 Test conditions

Vary minLocalDiff for the local feature descriptor in the conformance tests. At least the following values

for minLocalDiff should be tested: 1; 31.
5.2.3 Global descriptor test
Perform pairwise matching of global descriptors.

An implementation is considered conforming to the specification, if at least 99% of the descriptors of

the conformance dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.2.4 Local descriptor test
Perform pairwise matching of local descriptors.

An implementation is considered conforming to the specification if at least 99% of the descriptors of the

conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.2.5 Deep feature descriptor test with default NN
Perform pairwise matching of deep feature descriptors, using the default NN.

An implementation is considered conforming to the specification if at least 99% of the descriptors of the

conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.2.6 Combined global and local descriptor test
Perform pairwise matching combining local and global descriptors.

An implementation is considered conforming to the specification if at least 99% of the descriptors of the

conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

2 © ISO/IEC 2021 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 15938-16:2021(E)

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.2.7 Combined global and deep feature descriptor test with default NN

Perform pairwise matching combining local and global descriptors, using the default NN.

An implementation is considered conforming to the specification if at least 99% of the descriptors of the

conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.3 Conformance test conditions using custom NN

As a prerequisite for this test, at least the conformance tests for global descriptors shall be run.

No reference data is provided for this test.

Only syntactic checking can be performed by using the reference software to parse and match

bitstreams generated with the implementation under test. This shall complete without errors.

6 Reference software
6.1 Reference software location

The reference software is available at https:// standards .iso .org/ iso -iec/ 15938/ -16/ ed -1/ en.

6.2 Reference software licence

The licence of the software is described in the file CDVA_evaluation_framework/COPYING included with

the reference software.
6.3 Reference software documentation

The documentation of the software is contained in the folder CDVA_evaluation_framework/docs included

with the reference software. An overview is provided in the file CDVA_evaluation_framework/README.

6.4 Reference software installation and compilation

The installation of the software is described in file CDVA_evaluation_framework/INSTALLING included

with the reference software.
The CDVA reference s
...

FINAL
INTERNATIONAL ISO/IEC
DRAFT
STANDARD FDIS
15938-16
ISO/IEC JTC 1/SC 29
Information technology — Multimedia
Secretariat: JISC
content description interface —
Voting begins on:
2021-03-09
Part 16:
Voting terminates on:
Conformance and reference software
2021-05-04
for compact descriptors for video
analysis
RECIPIENTS OF THIS DRAFT ARE INVITED TO
SUBMIT, WITH THEIR COMMENTS, NOTIFICATION
OF ANY RELEVANT PATENT RIGHTS OF WHICH
THEY ARE AWARE AND TO PROVIDE SUPPOR TING
DOCUMENTATION.
IN ADDITION TO THEIR EVALUATION AS
Reference number
BEING ACCEPTABLE FOR INDUSTRIAL, TECHNO-
ISO/IEC FDIS 15938-16:2021(E)
LOGICAL, COMMERCIAL AND USER PURPOSES,
DRAFT INTERNATIONAL STANDARDS MAY ON
OCCASION HAVE TO BE CONSIDERED IN THE
LIGHT OF THEIR POTENTIAL TO BECOME STAN-
DARDS TO WHICH REFERENCE MAY BE MADE IN
NATIONAL REGULATIONS. ISO/IEC 2021
---------------------- Page: 1 ----------------------
ISO/IEC FDIS 15938-16:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC FDIS 15938-16:2021(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

Introduction ..................................................................................................................................................................................................................................v

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Abbreviated terms, relational operators, bitwise operators and functions ...........................................1

5 Conformance testing ......................................................................................................................................................................................... 1

5.1 Conformance video dataset and reference bitstreams ......................................................................................... 1

5.2 Conformance test conditions for strict conformance ............................................................................................ 2

5.2.1 Test setup ............................................................................................................................................................................... 2

5.2.2 Test conditions................................................................................................................................................................... 2

5.2.3 Global descriptor test .................................................................................................................................................. 2

5.2.4 Local descriptor test ..................................................................................................................................................... 2

5.2.5 Deep feature descriptor test with default NN ......................................................................................... 2

5.2.6 Combined global and local descriptor test ................................................................................................ 2

5.2.7 Combined global and deep feature descriptor test with default NN................................... 3

5.3 Conformance test conditions using custom NN ......................................................................................................... 3

6 Reference software ............................................................................................................................................................................................ 3

6.1 Reference software location ........................................................................................................................................................ 3

6.2 Reference software licence ........................................................................................................................................................... 3

6.3 Reference software documentation ...................................................................................................................................... 3

6.4 Reference software installation and compilation ..................................................................................................... 3

6.5 Reference software architecture .............................................................................................................................................. 3

6.5.1 General...................................................................................................................................................................................... 3

6.5.2 Extraction ............................................................................................................................................................................... 4

6.5.3 Pairwise matching .......................................................................................................................................................... 6

6.5.4 Retrieval .................................................................................................................................................................................. 7

6.5.5 Parameters ............................................................................................................................................................................ 8

6.5.6 Implementation details for CDVA reference software ..................................................................... 9

6.5.7 Usage .......................................................................................................................................................................................... 9

Annex A (informative) Deployment of the reference software using Docker containers...........................13

© ISO/IEC 2021 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC FDIS 15938-16:2021(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that

are members of ISO or IEC participate in the development of International Standards through

technical committees established by the respective organization to deal with particular fields of

technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other

international organizations, governmental and non-governmental, in liaison with ISO and IEC, also

take part in the work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives or www .iec .ch/ members

_experts/ refdocs).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC

list of patent declarations received (see patents.iec.ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/

iso/ foreword .html. In the IEC, see www .iec .ch/ understanding -standards.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.

A list of all parts in the ISO/IEC 15938 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/ members .html and www .iec .ch/ national

-committees.
iv © ISO/IEC 2021 – All rights reserved
---------------------- Page: 4 ----------------------
ISO/IEC FDIS 15938-16:2021(E)
Introduction

ISO/IEC 15938 (all parts) provides a standardized set of technologies for describing multimedia content.

It addresses a broad spectrum of multimedia applications and requirements by providing a metadata

system for describing the features of multimedia content.
The following are specified in ISO/IEC 15938 (all parts):

Description schemes (DS) describe entities or relationships pertaining to multimedia content.

Description schemes specify the structure and semantics of their components, which can be description

schemes, descriptors or datatypes.

Descriptors (D) describe features, attributes or groups of attributes of multimedia content.

Datatypes are the basic reusable datatypes employed by description schemes and descriptors.

Description definition language (DDL) defines description schemes, descriptors and datatypes by

specifying their syntax, and allows their extension.

Systems tools support delivery of descriptions, multiplexing of descriptions with multimedia content,

synchronization, file format, etc.

ISO 15938 (all parts) is subdivided into 16 published parts with further parts in development:

— Part 1: Systems: specifies the tools for preparing descriptions for efficient transport and storage,

compressing descriptions, and allowing synchronization between content and descriptions.

— Part 2: Description definition language: specifies the language for defining the series set of

description tools (DSs, Ds and datatypes) and for defining new description tools.

— Part 3: Visual: specifies the description tools pertaining to visual content.
— Part 4: Audio: specifies the description tools pertaining to audio content.

— Part 5: Multimedia description schemes: specifies the generic description tools pertaining to

multimedia including audio and visual content.
— Part 6: Reference software: provides a software implementation of the series.

— Part 7: Conformance testing: specifies the guidelines and procedures for testing conformance of

implementations of the series.

— Part 8: Extraction and use of MPEG-7 descriptions: provides guidelines and examples of the

extraction and use of descriptions.
— Part 9: Profiles and levels: provides guidelines and standard profiles.

— Part 10: Schema definition: specifies the schema using description definition language.

— Part 11: MPEG-7 profile schemas: listing of profile schemas using description definition language.

— Part 12: Query format: contains the tools of the MPEG query format (MPQF).

— Part 13: Compact descriptors for visual search: specifies an image description tool for visual

search applications.

— Part 14: Reference software, conformance and usage guidelines for compact descriptors for

visual search: provides the reference software and guidelines, specifies the conformance testing.

— Part 15: Compact descriptors for video analysis: specifies a video description tool designed to

enable efficient and interoperable video analysis applications, allowing visual content matching

in videos.
© ISO/IEC 2021 – All rights reserved v
---------------------- Page: 5 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

— Part 16 (this document): Conformance and reference software for compact descriptors for

video analysis: describes conformance testing of descriptors specified in ISO/IEC 15938-15, and

provides the reference software for extracting and matching such descriptors.
The structure of this document is as follows:
— Clause 5 specifies conformance testing of descriptors.

— Clause 6 describes the reference software for extracting and matching descriptors. The deployment

of the reference software using Docker containers is described in Annex A.
vi © ISO/IEC 2021 – All rights reserved
---------------------- Page: 6 ----------------------
FINAL DRAFT INTERNATIONAL STANDARD ISO/IEC FDIS 15938-16:2021(E)
Information technology — Multimedia content description
interface —
Part 16:
Conformance and reference software for compact
descriptors for video analysis
1 Scope

This document specifies the assessment of conformance to ISO/IEC 15938-15 as well as the reference

software.
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO/IEC 15938-14, Information technology — Multimedia content description interface — Part 14:

Reference software, conformance and usage guidelines for compact descriptors for visual search

ISO/IEC 15938-15, Information technology — Multimedia content description interface — Part 15:

Compact descriptors for video analysis
3 Terms and definitions

For the purposes of this document, the terms and definitions in ISO/IEC 15938-15 apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
4 Abbreviated terms, relational operators, bitwise operators and functions

For the purposes of this document, the abbreviated terms, relational operators, bitwise operators and

functions in ISO/IEC 15938-15 apply.
5 Conformance testing
5.1 Conformance video dataset and reference bitstreams

The CDVA conformance data set, containing 3 318 items, is used for conformance testing.

The data set is available at https:// standards .iso .org/ iso -iec/ 15938/ -16/ ed -1/ en.

For deep feature descriptors, two levels of conformance are considered:

— Conformance using default neural network: This is a stricter level of conformance, which ensures

that the bitstreams match those generated with the reference software as-is.
© ISO/IEC 2021 – All rights reserved 1
---------------------- Page: 7 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

— Conformance using custom neural network: This is a less strict level of conformance, that allows

using a custom neural network. The deep feature descriptors generated with implementations

using different neural networks are not interoperable.
5.2 Conformance test conditions for strict conformance
5.2.1 Test setup
— Bitstreams extracted with the reference software on Ubuntu 16.04 are provided.

— Implementation under test is used to extract bitstreams from the conformance test set, including all

three descriptor components, and varying the parameters listed in subclause 6.5.4.

— Use the reference implementation to perform pairwise matching between the bitstreams of

corresponding videos.

— Check that the resulting similarity is above the thresholds listed in the following subclauses.

5.2.2 Test conditions

Vary minLocalDiff for the local feature descriptor in the conformance tests. At least the following values

for minLocalDiff should be tested: 1; 31.
5.2.3 Global descriptor test
Perform pairwise matching of global descriptors.

An implementation is considered conforming to the specification, if at least 99% of the descriptors of

the conformance dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.2.4 Local descriptor test
Perform pairwise matching of local descriptors.

An implementation is considered conforming to the specification if at least 99% of the descriptors of the

conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.2.5 Deep feature descriptor test with default NN
Perform pairwise matching of deep feature descriptors, using the default NN.

An implementation is considered conforming to the specification if at least 99% of the descriptors of the

conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.2.6 Combined global and local descriptor test
Perform pairwise matching combining local and global descriptors.

An implementation is considered conforming to the specification if at least 99% of the descriptors of the

conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

2 © ISO/IEC 2021 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC FDIS 15938-16:2021(E)

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.2.7 Combined global and deep feature descriptor test with default NN

Perform pairwise matching combining local and global descriptors, using the default NN.

An implementation is considered conforming to the specification if at least 99% of the descriptors of the

conformance video dataset yield a score greater or equal to 0,9s, when matching bitstreams extracted

with the implementation under test with the reference bitstreams for the same video. s is the matching

score for the same pair of videos when matching the reference bitstreams to itself using the reference

implementation.
5.3 Conformance test conditions using custom NN

As a prerequisite for this test, at least the conformance tests for global descriptors shall be run.

No reference data is provided for this test.

Only syntactic checking can be performed by using the reference software to parse and match

bitstreams generated with the implementation under test. This shall complete without errors.

6 Reference software
6.1 Reference software location

The reference software is available at https:// standards .iso .org/ iso -iec/ 15938/ -16/ ed -1/ en.

6.2 Reference software licence

The licence of the software is described in the file CDVA_evaluation_framework/COPYING included with

the reference software.
6.3 Reference software documentation

The documentation of the software is contained in the folder CDVA_evaluation_framework/docs included

with the reference software. An overview is provided in the file CDVA_evaluation_framework/README.

6.4 Re
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.