ISO/IEC 15938-18:2023

Information technology — Multimedia content description interface — Part 18: Conformance and reference software for compression of neural networks

This document specifies conformance testing procedures for implementations of ISO/IEC 15938-17 and provides conformance bitstreams. It also provides the reference software for ISO/IEC 15938-17 which is an integral part of this document.

Technologies de l'information — Interface de description du contenu multimédia — Partie 18: Conformité et logiciel de référence pour la compression des réseaux neuronaux

General Information

• Status: Published
• Publication Date: 13-Jul-2023
• ICS: 35.040.40 - Coding of audio, video, multimedia and hypermedia information
• Technical Committee: ISO/IEC JTC 1/SC 29 - Coding of audio, picture, multimedia and hypermedia information
• Drafting Committee: ISO/IEC JTC 1/SC 29 - Coding of audio, picture, multimedia and hypermedia information
• Current Stage: 9092 - International Standard to be revised
• Completion Date: 14-Aug-2023

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 15938-18
First edition
2023-07
Information technology — Multimedia
content description interface —
Part 18:
Conformance and reference software
for compression of neural networks
Technologies de l'information — Interface de description du contenu
multimédia —
Partie 18: Conformité et logiciel de référence pour la compression des
réseaux neuronaux
Reference number
ISO/IEC 15938-18:2023(E)
© ISO/IEC 2023

---------------------- Page: 1 ----------------------
ISO/IEC 15938-18:2023(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2023
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
ii
  © ISO/IEC 2023 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC 15938-18:2023(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Conformance testing . .1
4.1 General . 1
4.2 Conformance testing for decoder . 2
4.3 Conformance testing for bitstreams . 2
4.4 Models and reference bitstreams . 2
4.5 Procedure to test decoders . 7
4.5.1 General . 7
4.5.2 Decoding self-contained NNC bitstreams . 8
4.5.3 Decoding NNC bitstreams using out-of-band parameters . 8
4.6 Procedure to test bitstreams . 8
5 Reference software . 8
5.1 General . 8
5.2 Software location and license . 9
5.3 Software installation . 9
5.4 Software architecture . 9
5.4.1 General . 9
5.4.2 Parameter reduction methods . 10
5.4.3 Parameter approximation . 10
5.4.4 Reconstruction . . . 10
5.4.5 Encode. 10
5.4.6 Decode . 11
5.5 Data structures and interfaces . 11
5.5.1 model_info: Shared model information . 11
5.5.2 approx_data – Data structure for interface #4 .12
5.5.3 nctm – Main module . 14
5.5.4 nctm.nnr_model – Module for handling model related functionalities . 17
Annex A (informative) Implementation in Python.21
Bibliography .22
iii
© ISO/IEC 2023 – All rights reserved

---------------------- Page: 3 ----------------------
ISO/IEC 15938-18:2023(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance
are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria
needed for the different types of document should be noted. This document was drafted in
accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or
www.iec.ch/members_experts/refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC
list of patent declarations received (see https://patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see
www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 15938 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.
iv
  © ISO/IEC 2023 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC 15938-18:2023(E)
Introduction
This document describes conformance testing and the reference software for ISO/IEC 15938-17
Compression of neural networks for multimedia content description and analysis. The reference
software includes both encoder and decoder functionality.
The reference software is useful in aiding users of a standard for coding neural networks to establish
and test conformance and interoperability, and to educate users and demonstrate the capabilities of
the standard. For these purposes, the accompanying software is provided as an aid for the study and
implementation of 15938-17 compression of neural networks for multimedia content description and
analysis.
The purpose of this document is to provide the following:
— A set of reference bitstreams conforming to ISO/IEC 15938-17.
— Description of procedures to test conformance of bitstreams and decoders to ISO/IEC 15938-17.
— Reference decoder software capable of decoding bitstreams that conform to ISO/IEC 15938-17 in a
manner that conforms to the decoding process specified in ISO/IEC 15938-17.
— Reference encoder software capable of producing bitstreams that conform to ISO/IEC 15938-17.
v
© ISO/IEC 2023 – All rights reserved

---------------------- Page: 5 ----------------------
INTERNATIONAL STANDARD ISO/IEC 15938-18:2023(E)
Information technology — Multimedia content description
interface —
Part 18:
Conformance and reference software for compression of
neural networks
1 Scope
This document specifies conformance testing procedures for implementations of ISO/IEC 15938-17 and
provides conformance bitstreams. It also provides the reference software for ISO/IEC 15938-17 which
is an integral part of this document.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 15938-17:2022, Information technology — Multimedia content description interface — Part 17:
Compression of neural networks for multimedia content description and analysis
ISO/IEC 21778, Information technology — The JSON data interchange syntax
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 15938-17 and the
following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
array
ordered list of elements where all elements are of the same type
3.2
dictionary
ordered list of key/value pairs where each key/value pair is an list of two elements with the first element
being denoted ‘key’ and with the second element being denoted ‘value’
4 Conformance testing
4.1 General
Subclauses 4.2 through 4.6 specify tests for verifying the conformance of bitstreams as well as decoders.
These tests make use of test data (bitstreams and related models) provided at https:// standards .iso
1
© ISO/IEC 2023 – All rights reserved

---------------------- Page: 6 ----------------------
ISO/IEC 15938-18:2023(E)
.org/ iso -iec/ 15938/ -18/ ed -1/ en (described in detail in 4.4) and follow the procedure described in
subclause 4.5.
4.2 Conformance testing for decoder
The decoder conformance is specified in ISO/IEC 15938-17:2022, Clause 7.
4.3 Conformance testing for bitstreams
The bitstream conformance is specified in ISO/IEC 15938-17:2022, Clause 7.
4.4 Models and reference bitstreams
A set of bitstreams and related neural network models is provided for conformance testing. When neural
network models are provided, they are the source models used to generate one or more compressed
bitstreams. The code defining these models is provided as part of the reference software distribution at
https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -1/ en. A bitstream can be generated:
— from a commonly used neural network model,
— from synthetic data by encoding it with the reference encoder,
— by creating a conformant synthetic bitstream directly, without using an encoder.
Where applicable, the dataset used to train the model is provided for information only. However, the
dataset is not needed for conformance testing and thus not provided.
Table 1 summarizes the provided bitstreams. It also lists the features tested with the bitstreams, and
the reference encoder configuration used to generate the bitstream (where applicable). If decoding the
bitstream requires out-of-band parameters (e.g. information that is derived from the network topology
description), those parameters are also provided. Source models and datasets are referred to using the
following names:
— ImageNet for the dataset described in Reference [5],
— CIFAR-100 for the dataset described in Reference [6],
— DCase for the dataset described in Reference [7] and the model described in Reference [8],
— MobileNet V2 for model described in Reference [9],
— UC12B for the model described in Reference [10] and
— VGG16 for the model described in Reference [11].
2
  © ISO/IEC 2023 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC 15938-18:2023(E)
Table 1 — Bitstreams and related models for conformance testing
Bitstream id Source Data-set Relevant Features test- Reference encoder configuration
model technology ed
ISO/IEC
15938-
17:2022
perf_map_ Mobile- Image 6.3.4.3 sparsification qp_density = 2
sparse_ NetV2 Net performance scan_order = 1
MobileNetV2. map approx_method = "codebook"
nctm qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_prune_ DCase DCase 6.3.4.3 pruning perfor- qp_density = 2
DCase.nctm mance map scan_order = 1
approx_method = "codebook"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_sparse_ UC12B 6.3.4.3 Sparsification qp_density = 2
prune_UC12B. and Pruning scan_order = 1
nctm Performance approx_method = "codebook"
map qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_ VGG16 Image 6.3.4.3 Sparsification qp_density = 2
sparse_VGG16. Net Performance scan_order = 1
nctm Map (pruned approx_method = "codebook"
model) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
prune_tpl_cont_ DCase DCase 6.3.4.5 Prune Topol-
encode_tpl_only = True
sparse_bm_ ogy - sparse
partial_data_counter = 0
DCase.nctm bitmask
3
© ISO/IEC 2023 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC 15938-18:2023(E)
TTabablele 1 1 ((ccoonnttiinnueuedd))
Bitstream id Source Data-set Relevant Features test- Reference encoder configuration
model technology ed
ISO/IEC
15938-
17:2022
prune_tpl_cont_ VGG16 Image 6.3.4.5 Prune Topology
encode_tpl_only = True
prune_bm_ Net - prune bitmask
partial_data_counter = 0
VGG16.nctm
prune_tpl_cont_ VGG16 Image 6.3.4.5 Prune Topolo-
encode_tpl_only = True
comb_bm_ Net gy - combined
partial_data_counter = 0
VGG16.nctm bitmask
prune_tpl_cont_ DCase DCase 6.3.4.5 Prune Topol- encode_tpl_only = True
prune_dictionary_ ogy - prune topology_indexed_reference_flag =
DCase.nctm dictionary False
partial_data_counter = 0
prune_tpl_cont_ ResNet50 Image 6.3.4.5 Prune Topology encode_tpl_only = True
prune_ Net - prune diction- topology_indexed_reference_flag =
dictionary_idx_ ary (indexed True
ResNet50.nctm elem id) partial_data_counter = 0
tpl_reflist_DCase. DCase DCase 6.3.4.5, Topology Reflist encode_tpl_only = True
nctm 6.3.3.7 partial_data_counter = 0
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter
max_ndu_nnr_unit_size = 65536
ndu_size_65536.
nctm
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter
max_ndu_nnr_unit_size = 32768
ndu_size_32768.
nctm
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter
max_ndu_nnr_unit_size = 16384
ndu_size_16384.
nctm
partial_data_ DCase DCase 6.3.3.1 Partial data
counter_DCase_ counter
max_ndu_nnr_unit_size = 2048
ndu_size_2048.
nctm
partial_data_coun- DCase DCase 6.3.3.1 Partial data
ter_DCase_ndu_ counter max_ndu_nnr_unit_size = 1024
size_1024.nctm
deepCABAC_ ResNet50 Image 10 DeepCABAC see verify_all.sh
ResNet50_1_ Net 9.1.1 / 9.2.1 entropy coding,
qp-38_qp_densi- uniform quanti-
ty2.nctm zation
dependent_ ResNet50 Image 9.1.3 / 9.2.3 Dependent sca- see verify_all.sh
quantization_ Net 6.3.3.7 lar quantization
ResNet50_2_
qp-38_qp_densi-
ty2.nctm
deepCABAC_qp_ Mobile- Image 9.2 QpDensity see verify_all.sh
density_Mobile- NetV2 Net
NetV2_3_qp-38_
qp_density2.nctm
4
  © ISO/IEC 2023 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/IEC 15938-18:2023(E)
TTabablele 1 1 ((ccoonnttiinnueuedd))
Bitstream id Source Data-set Relevant Features test- Reference encoder configuration
model technology ed
ISO/IEC
15938-
17:2022
deepCABAC_ Mobile- Image 9.2 QpDensity see verify_all.sh
qp_density_ NetV2 Net
MobileNetV2_4_
qp-76_qp_densi-
ty3.nctm
block_scan_ ResNet50 Image 4.12 / 6.4.3.7 Block scan see verify_all.sh
order_8x8_cabac_ Net / 6.4.3.8 / order / cabac
entry_points_ 7.3.6 entry points
ResNet50_5_
qp-38_qp_densi-
ty2.nctm
block_scan_or- ResNet50 Image 4.12 / 6.4.3.7 Block scan see verify_all.sh
der_16x16_cabac_ Net / 6.4.3.8 / order / cabac
entry_points_ 7.3.6 entry points
ResNet50_6_
qp-38_qp_densi-
ty2.nctm
codebook_sig- Mobile- Image 9.1.3 / 9.2.2 Codebook-based see verify_all.sh
naling_Mo- NetV2 Net quantization
bileNetV2_7_
qMse0.00001.
nctm
local_scaling_ DCase DCase 8.2.7 / 8.3.7 Local scaling see verify_all.sh
DCase_8_qp-38_
qp_density2.nctm
batchnorm_ Mobile- Image 8.2.6 / 8.3.6 BatchNorm see verify_all.sh
folding_Mobile NetV2 Net Folding
NetV2_9_qp-38_
qp_density2.nctm
out_of_band_ ResNet50 Image 6.3.3.7 / Out-of-band see verify_all.sh
a
signaling_Res Net 6.4.3.7 signaling
Net50_10_qp-38_
qp_density2.nctm
deepCABAC_8bit_ ResNet50 Image 9.1.1 / 9.2.1 Uniform quanti- see verify_all.sh
ResNet50_PYT- Net zation with lim-
zoo_11_qp0_qp_ ited precision
density4.pt.nctm (8bit)
deepCABAC_8bit_ MobileNet Image 9.1.1 / 9.2.1 Uniform quanti- see verify_all.sh
MobileNetV2_PY- V2 Net zation with lim-
Tzoo_12_qp0_qp_ ited precision
density4.pt.nctm (8bit)
deepCABAC_4bit_ VGG16 Image 9.1.1 / 9.2.1 Uniform quanti- see verify_all.sh
VGG16_PYT- Net zation with lim-
zoo_13_qp0_qp_ ited precision
density4.pt.nctm (4bit)
deepCABAC_8bit_ UC12B CIFAR- 9.1.1 / 9.2.1 Uniform quanti- see verify_all.sh
UC12B_14_qp0_ 100 zation with lim-
qp_density4.nctm ited precision
(8bit)
5
© ISO/IEC 2023 – All rights reserved

---------------------- Page: 10 ----------------------
ISO/IEC 15938-18:2023(E)
TTabablele 1 1 ((ccoonnttiinnueuedd))
Bitstream id Source Data-set Relevant Features test- Reference encoder configuration
model technology ed
ISO/IEC
15938-
17:2022
deepCABAC_4bit_ DCase DCase 9.1.1 / 9.2.1 Uniform quanti- see verify_all.sh
DCase_15_qp0_ zation with lim-
qp_density4.nctm ited precision
(4bit)
perf_map_ Mobile- Image 6.3.4.3 sparsification qp_density = 2
sparse_ NetV2 Net performance scan_order = 1
MobileNetV2_bw8. map (8bit) approx_method = "uniform"
nctm qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_prune_ DCase DCase 6.3.4.3 pruning per- qp_density = 2
DCase_bw8.nctm formance map scan_order = 1
(8bit) approx_method = "uniform"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_sparse_ UC12B 6.3.4.3 Sparsification qp_density = 2
prune_UC12B_ and Pruning scan_order = 1
bw8.nctm Performance approx_method = "uniform"
map (8bit) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
6
  © ISO/IEC 2023 – All rights reserved

---------------------- Page: 11 ----------------------
ISO/IEC 15938-18:2023(E)
TTabablele 1 1 ((ccoonnttiinnueuedd))
Bitstream id Source Data-set Relevant Features test- Reference encoder configuration
model technology ed
ISO/IEC
15938-
17:2022
perf_map_ VGG16 Image 6.3.4.3 Sparsification qp_density = 2
sparse_VGG16_ Net Performance scan_order = 1
bw8. Map (pruned approx_method = "uniform"
nctm model) (8bit) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
partial_data_ DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size = 2048
counter_DCase_ counter (8bit)
ndu_size_2048_
approx_method = "uniform"
bw8.nctm
partial_data_coun- DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size = 1024
ter_DCase_ndu_ counter (8bit)
size_1024_bw8.
approx_method = "uniform"
nctm
a
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
The bitstreams and models are provided at https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -1/ en/ .
The parameters of the bitstreams, for which no detailed parameters are listed, can be obtained from
the scripts given in column “Reference encoder configuration”, for example, verify_all.sh in “run/
conformance_bitstreams” of the reference software. Furthermore, it also provides detailed reference
software configurations for each test. If out-of-band parameters are required, they are provided in a
file conforming to ISO/IEC 21778 (JSON), specified in a footnote.
4.5 Procedure to test decoders
4.5.1 General
For testing a decoder, all bitstreams that cover mandatory features as specified in ISO/IEC 15938-17
shall be tested. If support for optional features is claimed, also the bitstreams covering these features
shall be tested.
On platforms without floating point support, or limited integer precision support, only those bitstreams
indicated as "(8bit)" or "(4bit)" shall be tested. On platforms with floating point support, testing these
limited precision bitstreams is optional.
For each bitstream, the procedure includes the following steps:
1) Decode bitstream B with the reference decoder to obtain the decoded model R according to
r
subclause 4.5.2 if no out-of-band parameters are specified and according to subclause 4.5.3,
otherwise,
2) Decode bitstream B with the decoder under test to obtain the decoded model R according to
t
subclause 4.5.2 if no out-of-band parameters are specified and according to subclause 4.5.3,
otherwise.
7
© ISO/IEC 2023 – All rights reserved

---------------------- Page: 12 ----------------------
ISO/IEC 15938-18:2023(E)
3) Compare every tensor in R against the corresponding tensor in R . The test shall be considered as
r t
passed if the following conditions hold for every pair of tensors:
— If an integer tensor has been obtained as a result (decoding method specified in
ISO/IEC 15938-17:2022, 7.3.2), the tensors shall be identical.
— If a floating-point tensor has been obtained as the result of another decoding method, no
corresponding pair of floating point values shall have an absolute difference larger than
0,000001.
4.5.2 Decoding self-contained NNC bitstreams
For self-contained bitstreams, the decoder shall not use any other inputs than the bitstream to be
decoded.
4.5.3 Decoding NNC bitstreams using out-of-band parameters
The testing of decoding bitstreams using out-of-band parameters is agnostic to whether those are
application provided or decoded from a particular third party topology format. The decoder shall use
only the input bitstream and the parameters provided for this bitstream as specified in Table 1 as
inputs.
4.6 Procedure to test bitstreams
In order to test bitstreams created with an encoder other than the reference encoder, the bitstream
shall be decoded using the reference decoder. The bitstream is considered conformant, if no errors
occur during the decoding process.
All bitstreams that have been created with the script “verify_all.sh” specified in Table 1 can be decoded
with the script “decode_all.sh” in “run/conformance_bitstreams” of the reference software. The
bitstreams are considered conformant, if no errors occur during the decoding process.
5 Reference software
5.1 General
The purpose of this clause is to provide the following:
— Reference decoder software capable of decoding bitstreams that conform to ISO/IEC 15938-17 in a
manner that conforms to the decoding process specified in ISO/IEC 15938-17.
— Reference encoder software capable of producing bitstreams that conform to ISO/IEC 15938-17.
Some examples of uses that may be appropriate for the reference decoder software are as follows:
— As an illustration of how to perform the decoding process specified in ISO/IEC 15938-17.
— As the starting basis for the implementation of a decoder that conforms to ISO/IEC 15938-17.
— For testing the conformance of a decoder implementation with the decoding process specified in
ISO/IEC 15938-17.
— For testing the conformance of a bitstream to the constraints specified for bitstream conformance
in ISO/IEC 15938-17, as the software can detect and report many bitstream conformance violations.
However, the lack of the detection of any conformance violation by the reference decoder software
should not be considered as definitive proof that the bitstream conforms to all constraints specified for
bitstream conformance in ISO/IEC 15938-17.
8
  © ISO/IEC 2023 – All rights reserved

---------------------- Page: 13 ----------------------
ISO/IEC 15938-18:2023(E)
Some examples of uses that may be appropriate for the reference encoder software are as follows:
— As an illustration of how to perform an encoding process that produces bitstreams that conform to
the constraints specified for bitstream conformance in ISO/IEC 15938-17.
— As the starting basis for the implementation of an encoder that conforms to ISO/IEC 15938-17.
— As a means of generating bitstreams for testing the conformance of a decoder implementation with
the decoding process specified in ISO/IEC 15938-17.
— As a means of evaluating and demonstrating examples of the quality that can be achieved by an
encoding process that conforms to ISO/IEC 15938-17.
5.2 Software location and license
The reference software is available at https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -1/ en/ .
The software license is provided in the file LICENSE included with the software.
5.3 Software installation
Installation instructions are provided in the file Readme.md included with the software.
5.4 Software architecture
5.4.1 General
Figure 1 gives a general overview of the discussed modules that are commonly present in the end-to-
end neural network compression pipeline. Each of the grey boxes specifies a module representing a
type of compression tools. Therefore, the interfaces that define the in
...

DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 15938-18
ISO/IEC JTC 1/SC 29 Secretariat: JISC
Voting begins on: Voting terminates on:
2022-07-25 2022-10-17
Information technology — Multimedia content description
interface —
Part 18:
Conformance and reference software for compression of
neural networks
ICS: 35.040.40
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.
NOT BE REFERRED TO AS AN INTERNATIONAL
STANDARD UNTIL PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
WHICH REFERENCE MAY BE MADE IN
Reference number
NATIONAL REGULATIONS.
ISO/IEC DIS 15938-18:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION. © ISO/IEC 2022

---------------------- Page: 1 ----------------------
ISO/IEC DIS 15938-18:2022(E)
DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 15938-18
ISO/IEC JTC 1/SC 29 Secretariat: JISC
Voting begins on: Voting terminates on:

Information technology — Multimedia content description
interface —
Part 18:
Conformance and reference software for compression of
neural networks
ICS: 35.040.40
COPYRIGHT PROTECTED DOCUMENT
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
© ISO/IEC 2022
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
NOT BE REFERRED TO AS AN INTERNATIONAL
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on STANDARD UNTIL PUBLISHED AS SUCH.
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
IN ADDITION TO THEIR EVALUATION AS
or ISO’s member body in the country of the requester. BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
ISO copyright office
USER PURPOSES, DRAFT INTERNATIONAL
CP 401 • Ch. de Blandonnet 8
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
CH-1214 Vernier, Geneva
POTENTIAL TO BECOME STANDARDS TO
Phone: [phone]
WHICH REFERENCE MAY BE MADE IN
Reference number
Email: [e-mail]
NATIONAL REGULATIONS.
Website: www.iso.org ISO/IEC DIS 15938-18:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
Published in Switzerland
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
ii
  © ISO/IEC 2022 – All rights reserved
PROVIDE SUPPORTING DOCUMENTATION. © ISO/IEC 2022

---------------------- Page: 2 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Conformance testing . .2
4.1 General . 2
4.2 Conformance testing for decoder . 2
4.3 Conformance testing for bitstreams . 2
4.4 Models and reference bitstreams . 2
4.5 Procedure to test decoders . 7
4.5.1 Decoding self-contained NNR bitstreams . 7
4.5.2 Decoding NNR bitstreams using out-of-band parameters . 7
4.6 Procedure to test bitstreams . 7
5 Reference software . 7
5.1 Software location and license . 7
5.2 Software installation . 8
5.3 Software architecture . 8
5.3.1 General . 8
5.3.2 Parameter reduction methods . 9
5.3.3 Parameter approximation . 9
5.3.4 Reconstruction . . . 10
5.3.5 Encode. 10
5.3.6 Decode . 10
5.4 Data structures and interfaces . 10
5.4.1 model_info – Shared model information. 10
5.4.2 approx_data – Data structure for interface #4 .12
5.4.3 nctm – Main module .13
5.4.4 nctm.nnr_model – Module for handling model related functionalities . 16
Annex A (informative) Implementation in Python .20
Bibliography .21
iii
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 3 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see
www.iso.org/iso/foreword.html.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 15938 series can be found on the ISO website and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www.iso.org/members.html.
iv
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Introduction
This document describes conformance testing and the reference software for ISO/IEC 15938-17
compression of neural networks for multimedia description and analysis. The reference software
includes both encoder and decoder functionality.
The reference software is useful in aiding users of a standard for coding neural networks to establish
and test conformance and interoperability, and to educate users and demonstrate the capabilities of
the standard. For these purposes, the accompanying software is provided as an aid for the study and
implementation of 15938-17 compression of neural networks for multimedia description and analysis.
The purpose of International Standard is to provide the following:
— A set of reference bitstreams conforming to ISO/IEC 15938-17.
— Description of procedures to test conformance of bitstreams and decoders to ISO/IEC 15938-17.
— Reference decoder software capable of decoding bitstreams that conform to ISO/IEC 15938-17 in a
manner that conforms to the decoding process specified in ISO/IEC 15938-17.
— Reference encoder software capable of producing bitstreams that conform to ISO/IEC 15938-17.
v
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 5 ----------------------
DRAFT INTERNATIONAL STANDARD ISO/IEC DIS 15938-18:2022(E)
Information technology — Multimedia content description
interface —
Part 18:
Conformance and reference software for compression of
neural networks
1 Scope
This document specifies conformance testing procedures for implementations of ISO/IEC 15938-17 and
provides conformance bitstreams. It also provides the reference software for ISO/IEC 15938-17 as an
electronic attachment. The software is an integral part of this International Standard.
Requirements established in ISO/IEC 15938-17 take precedence over the behaviour of the reference
software. The use of this reference software is not required for making an implementation of an
encoder or decoder in conformance to ISO/IEC 15938-17. The reference software is however required
for the conformance testing procedures of decoders and bitstreams.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 15938-17:2022, Information technology — Multimedia content description interface — Part 17:
Compression of neural networks for multimedia content description and analysis
ISO/IEC 21778, Information technology — The JSON data interchange syntax
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 15938-17 and the
following apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
array
ordered list of elements where all elements are of the same type
3.2
dictionary
ordered list of key/value pairs where each key/value pair is an list of two elements with the first element
being denoted ‘key’ and with the second element being denoted ‘value’
1
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 6 ----------------------
ISO/IEC DIS 15938-18:2022(E)
4 Conformance testing
4.1 General
The following subclauses specify normative tests for verifying the conformance of bitstreams as well as
decoders. Those normative tests make use of test data (bitstreams and related models) provided as an
electronic attachment to this document (described in detail in subclause 4.4) and follow the procedure
described in subclause 4.5.
4.2 Conformance testing for decoder
The decoder conformance of ISO/IEC 15938-17 is specified in clause 7 of that document.
4.3 Conformance testing for bitstreams
The bitstream conformance of ISO/IEC 15938-17 is specified in clause 7 of that document.
4.4 Models and reference bitstreams
A set of bitstreams and related neural network models is provided for conformance testing. When neural
network models are provided, they are the source models used to generate one or more compressed
bitstreams. The code defining these models is provided as part of the reference software distribution. A
bitstream may be generated
— from a commonly used neural network model, e.g., MobileNet
— from synthetic data by encoding it with the reference encoder
— by creating a conformant synthetic bitstream directly, without using an encoder
Where applicable, the dataset used to train the model is provided for information only. However, the
dataset is not needed for conformance testing and thus not provided.
Table 1 summarizes the provided bitstreams. It also lists the features tested with the bitstreams, and
the reference encoder configuration used to generate the bitstream (where applicable). If decoding the
bitstream requires out-of-band parameters (e.g., information that is derived from the network topology
description), those parameters are also provided.
Table 1 — Bitstreams and related models for conformance testing.
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
perf_map_ MobileNetV2 Image 6.3.4.3 sparsification qp_density = 2
sparse_ Net performance map scan_order = 1
MobileNetV2. approx_method = "codebook"
nctm qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
2
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Table 1 (continued)
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
perf_map_prune_ DCase DCase 6.3.4.3 pruning perfor- qp_density = 2
DCase.nctm mance map scan_order = 1
approx_method = "codebook"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
perf_map_sparse_ UC12B 6.3.4.3 Sparsification qp_density = 2
prune_UC12B.nctm and Pruning Per- scan_order = 1
formance map approx_method = "codebook"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
perf_map_ VGG16 Image 6.3.4.3 Sparsification qp_density = 2
sparse_VGG16. Net Performance Map scan_order = 1
nctm (pruned model) approx_method = "codebook"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
prune_tpl_cont_ DCase DCase 6.3.4.5 Prune Topology -
encode_tpl_only = True
sparse_bm_ sparse bitmask
partial_data_counter = 0
DCase.nctm
prune_tpl_cont_ VGG16 Image 6.3.4.5 Prune Topology -
encode_tpl_only = True
prune_bm_ Net prune bitmask
partial_data_counter = 0
VGG16.nctm
prune_tpl_cont_ VGG16 Image 6.3.4.5 Prune Topolo-
encode_tpl_only = True
comb_bm_ Net gy - combined
partial_data_counter = 0
VGG16.nctm bitmask
prune_tpl_cont_ DCase DCase 6.3.4.5 Prune Topology - encode_tpl_only = True
prune_dictionary_ prune dictionary topology_indexed_reference_
DCase.nctm flag = False
partial_data_counter = 0
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
3
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Table 1 (continued)
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
prune_tpl_cont_ ResNet50 Image 6.3.4.5 Prune Topology - encode_tpl_only = True
prune_ Net prune dictionary topology_indexed_reference_
dictionary_idx_ (indexed elem id) flag = True
ResNet50.nctm partial_data_counter = 0
tpl_reflist_DCase. DCase DCase 6.3.4.5, Topology Reflist encode_tpl_only = True
nctm 6.3.3.7 partial_data_counter = 0
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter max_ndu_nnr_unit_size =
ndu_size_65536. 65536
nctm
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter max_ndu_nnr_unit_size =
ndu_size_32768. 32768
nctm
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter max_ndu_nnr_unit_size =
ndu_size_16384. 16384
nctm
partial_data_ DCase DCase 6.3.3.1 Partial data
counter_DCase_ counter max_ndu_nnr_unit_size =
ndu_size_2048. 2048
nctm
partial_data_coun- DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size =
ter_DCase_ndu_ counter
1024
size_1024.nctm
deepCABAC_ ResNet50 Image 10 DeepCABAC en- see verify_all.sh
ResNet50_1_qp-38_ Net 9.1.1 / 9.2.1 tropy coding,
qp_density2.nctm uniform quanti-
zation
dependent_ ResNet50 Image 9.1.3 / 9.2.3 Dependent scalar see verify_all.sh
quantization_ Net 6.3.3.7 quantization
ResNet50_2_qp-38_
qp_density2.nctm
deepCABAC_qp_ MobileNetV2 Image 9.2 QpDensity see verify_all.sh
density_Mobile- Net
NetV2_3_qp-38_qp_
density2.nctm
deepCABAC_ MobileNetV2 Image 9.2 QpDensity see verify_all.sh
qp_density_ Net
MobileNetV2_4_
qp-76_qp_density3.
nctm
block_scan_ ResNet50 Image 4.12 / 6.4.3.7 Block scan order see verify_all.sh
order_8x8_cabac_ Net / 6.4.3.8 / / cabac entry
entry_points_ 7.3.6 points
ResNet50_5_qp-38_
qp_density2.nctm
block_scan_or- ResNet50 Image 4.12 / 6.4.3.7 Block scan order see verify_all.sh
der_16x16_cabac_ Net / 6.4.3.8 / / cabac entry
entry_points_ 7.3.6 points
ResNet50_6_qp-38_
qp_density2.nctm
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
4
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Table 1 (continued)
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
codebook_signal- MobileNetV2 Image 9.1.3 / 9.2.2 Codebook-based see verify_all.sh
ing_MobileNetV2_7_ Net quantization
qMse0.00001.nctm
local_scaling_ DCase DCase 8.2.7 / 8.3.7 Local scaling see verify_all.sh
DCase_8_qp-38_qp_
density2.nctm
batchnorm_ MobileNetV2 Image 8.2.6 / 8.3.6 BatchNorm see verify_all.sh
folding_Mobile Net Folding
NetV2_9_qp-38_qp_
density2.nctm
out_of_band_ ResNet50 Image 6.3.3.7 / Out-of-band sign- see verify_all.sh
2
signaling_Res Net 6.4.3.7 aling
Net50_10_qp-38_
qp_density2.nctm
deepCABAC_8bit_ resnet50 ImageNet 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
ResNet50_PYT- (zoo) tization with
zoo_11_qp0_qp_ limited precision
density4.pt.nctm (8bit)
deepCABAC_8bit_ mobilenet_v2 ImageNet 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
MobileNetV2_PY- (zoo) tization with
Tzoo_12_qp0_qp_ limited precision
density4.pt.nctm (8bit)
deepCABAC_4bit_ vgg16 (zoo) ImageNet 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
VGG16_PYTzoo_13_ tization with
qp0_qp_density4. limited precision
pt.nctm (4bit)
deepCABAC_8bit_ UC12B CIFAR- 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
UC12B_14_qp0_qp_ 100 tization with
density4.nctm limited precision
(8bit)
deepCABAC_4bit_ DCase DCase 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
DCase_15_qp0_qp_ tization with
density4.nctm limited precision
(4bit)
perf_map_ MobileNetV2 Image 6.3.4.3 sparsification qp_density = 2
sparse_ Net performance map scan_order = 1
MobileNetV2_bw8. (8bit) approx_method = "uniform"
nctm qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
5
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 10 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Table 1 (continued)
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
perf_map_prune_ DCase DCase 6.3.4.3 pruning perfor- qp_density = 2
DCase_bw8.nctm mance map (8bit) scan_order = 1
approx_method = "uniform"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
perf_map_sparse_ UC12B 6.3.4.3 Sparsification qp_density = 2
prune_UC12B_bw8. and Pruning scan_order = 1
nctm Performance map approx_method = "uniform"
(8bit) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
perf_map_ VGG16 Image 6.3.4.3 Sparsification qp_density = 2
sparse_VGG16_bw8. Net Performance Map scan_order = 1
nctm (pruned model) approx_method = "uniform"
(8bit) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
partial_data_ DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size =
counter_DCase_ counter (8bit)
2048
ndu_size_2048_
approx_method = "uniform"
bw8.nctm
partial_data_coun- DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size =
ter_DCase_ndu_ counter (8bit)
1024
size_1024_bw8.
approx_method = "uniform"
nctm
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
The bitstreams and models are provided at https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -1/ en/ .
The parameters of the bitstreams, for which no detailed parameters are listed, can be obtained from
the scripts given in column “Reference encoder configuration”, e.g., verify_all.sh in “run/conformance_
bitstreams” of the reference software. Furthermore, it also provides detailed reference software
configurations for each test. If out-of-band parameters are required, they are provided in a file
conforming to JSON (ISO/IEC 21778), specified in a footnote.
6
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 11 ----------------------
ISO/IEC DIS 15938-18:2022(E)
4.5 Procedure to test decoders
For testing a decoder, all bitstreams that cover mandatory features as specified in ISO/IEC 15938-17
shall be tested. If support for optional features is claimed, also the bitstreams covering these features
shall be tested.
On platforms without floating point support, or limited integer precision support, only those bitstreams
indicated as "(8bit)" or "(4bit)" shall be tested. On platforms with floating point support, testing these
limited precision bitstreams is optional.
For each bitstream, the procedure includes the following steps:
1. Decode bitstream B with the reference decoder to obtain the decoded model R according to
r
subclause 4.5.1 if no out-of-band parameters are specified and according to subclause 4.5.2,
otherwise,
2. Decode bitstream B with the decoder under test to obtain the decoded model R according to
t
subclause 4.5.1 if no out-of-band parameters are specified and according to subclause 4.5.2,
otherwise.
3. Compare every tensor in R against the corresponding tensor in R . The test shall be considered as
r t
passed if the following conditions hold for every pair of tensors:
— If an integer tensor has been obtained as a result (decoding method specified in clause 7.3.2 of
ISO/IEC 15938-17), the tensors shall be identical.
— If a tensor has been obtained as the result of another decoding method, no corresponding pair
of floating point values shall have an absolute difference larger than 0,000001.
4.5.1 Decoding self-contained NNR bitstreams
For self-contained bitstreams, the decoder may not use any other inputs than the bitstream to be
decoded.
4.5.2 Decoding NNR bitstreams using out-of-band parameters
The testing of decoding bitstreams using out-of-band parameters is agnostic to whether those are
application provided or decoded from a particular third party topology format. The decoder shall use
only the input bitstream and the parameters provided for this bitstream as specified in Table 1 as
inputs.
4.6 Procedure to test bitstreams
In order to test bitstreams created with an encoder other than the reference encoder, the bitstream
shall be decoded using the reference decoder. The bitstream is considered conformant, if no errors
occur during the decoding process.
All bitstreams that have been created with the script “verify_all.sh” specified in Table 1 can be decoded
with the script “decode_all.sh” in “run/conformance_bitstreams” of the reference software. The
bitstreams are considered conformant, if no errors occur during the decoding process.
5 Reference software
5.1 Software location and license
The reference software is available at https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -1/ en/ .
The software license is provided in the file LICENSE included with the software.
7
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 12 ----------------------
ISO/IEC DIS 15938-18:2022(E)
The purpose of this clause is to provide the following:
— Reference decoder software capable of decoding bitstreams that conform to ISO/IEC 15938-17 in a
manner that conforms to the decoding process specified in ISO/IEC 15938-17.
— Reference encoder software capable of producing bitstreams that conform to ISO/IEC 15938-17.
Some examples of uses that may be appropriate for the reference decoder software are as follows:
— As an illustration of how to perform the decoding process specified in ISO/IEC 15938-17.
— As the starting basis for the implementation of a decoder that conforms to ISO/IEC 15938-17.
— For testing the conformance of a decoder implementation with the decoding process specified in
ISO/IEC 15938-17.
— For testing the conformance of a bitstream to the constraints specified for bitstream conformance
in ISO/IEC 15938-
...