ISO/IEC 15938-18:2025

Information technology — Multimedia content description interface — Part 18: Conformance and reference software for compression of neural networks

This document specifies conformance testing procedures for implementations of ISO/IEC 15938-17 and provides conformance bitstreams. It also provides the reference software for ISO/IEC 15938-17 which is an integral part of this document.

Technologies de l'information — Interface de description du contenu multimédia — Partie 18: Conformité et logiciel de référence pour la compression des réseaux neuronaux

General Information

• Status: Published
• Publication Date: 25-Feb-2025
• ICS: 35.040.40 - Coding of audio, video, multimedia and hypermedia information
• Technical Committee: ISO/IEC JTC 1/SC 29 - Coding of audio, picture, multimedia and hypermedia information
• Drafting Committee: ISO/IEC JTC 1/SC 29 - Coding of audio, picture, multimedia and hypermedia information
• Current Stage: 6060 - International Standard published
• Start Date: 26-Feb-2025
• Due Date: 14-Feb-2025
• Completion Date: 26-Feb-2025

Relations

• Revises: ISO/IEC 15938-18:2023 - Information technology — Multimedia content description interface — Part 18: Conformance and reference software for compression of neural networks
• Effective Date: 19-Aug-2023

Standards Content (Sample)

International
Standard
ISO/IEC 15938-18
Second edition
Information technology —
2025-02
Multimedia content description
interface —
Part 18:
Conformance and reference
software for compression of neural
networks
Technologies de l'information — Interface de description du
contenu multimédia —
Partie 18: Conformité et logiciel de référence pour la compression
des réseaux neuronaux
Reference number
© ISO/IEC 2025
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2025 – All rights reserved
ii
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Conformance testing . . 1
4.1 General .1
4.2 Conformance testing for decoder .2
4.3 Conformance testing for bitstreams .2
4.4 Models and reference bitstreams .2
4.5 Procedure to test decoders .10
4.5.1 General .10
4.5.2 Complete model .10
4.5.3 Incremental model .11
4.5.4 Decoding self-contained NNC bitstreams .11
4.5.5 Decoding NNC bitstreams using out-of-band parameters .11
4.6 Procedure to test bitstreams .11
5 Reference software .11
5.1 General .11
5.2 Software location and license . 12
5.3 Software installation . 12
5.4 Software architecture . 12
5.4.1 General . 12
5.4.2 Parameter reduction methods . 13
5.4.3 Parameter approximation .14
5.4.4 Reconstruction . . .14
5.4.5 Encode.14
5.4.6 Decode .14
5.5 Data structures and interfaces .14
5.5.1 model_info: Shared model information .14
5.5.2 approx_data – Data structure .16
5.5.3 nnc_core – Main module .18
5.5.4 nnc_core.nnr_model – Module for handling model related functionalities . 22
Annex A (informative) Implementation in Python®.25
Bibliography .26

© ISO/IEC 2025 – All rights reserved
iii
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical activity.
ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations,
governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types
of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/
IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any
claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held
responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html.
In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
This second edition cancels and replaces the first edition (ISO/IEC 15938-18:2023), which has been
technically revised.
The main changes are:
— additional conformance bitstreams and an extension of the reference software to cover the features
added in ISO/IEC 15938-17:2024.
A list of all parts in the ISO/IEC 15938 series can be found on the ISO website and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.

© ISO/IEC 2025 – All rights reserved
iv
Introduction
This document describes conformance testing and the reference software for ISO/IEC 15938-17. The
reference software includes both encoder and decoder functionality.
The reference software is useful in aiding users of a standard for coding neural networks to establish and test
conformance and interoperability, and to educate users and demonstrate the capabilities of the standard.
For these purposes, the accompanying software is provided as an aid for the study and implementation of
ISO/IEC 15938-17.
The purpose of this document is to provide the following:
— A set of reference bitstreams conforming to ISO/IEC 15938-17.
— Description of procedures to test conformance of bitstreams and decoders to ISO/IEC 15938-17.
— Reference decoder software capable of decoding bitstreams that conform to ISO/IEC 15938-17 in a
manner that conforms to the decoding process specified in ISO/IEC 15938-17.
— Reference encoder software capable of producing bitstreams that conform to ISO/IEC 15938-17.

© ISO/IEC 2025 – All rights reserved
v
International Standard ISO/IEC 15938-18:2025(en)
Information technology — Multimedia content description
interface —
Part 18:
Conformance and reference software for compression of
neural networks
1 Scope
This document specifies conformance testing procedures for implementations of ISO/IEC 15938-17 and
provides conformance bitstreams. It also provides the reference software for ISO/IEC 15938-17 which is an
integral part of this document.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content constitutes
requirements of this document. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 15938-17:2024, Information technology — Multimedia content description interface — Part 17:
Compression of neural networks for multimedia content description and analysis
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 15938-17 and the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
array
ordered list of elements where all elements are of the same type
3.2
dictionary
ordered list of key/value pairs where each key/value pair is a list of two elements with the first element
being denoted ‘key’ and with the second element being denoted ‘value’
4 Conformance testing
4.1 General
Subclauses 4.2 through 4.6 specify tests for verifying the conformance of bitstreams as well as decoders.
These tests make use of test data (bitstreams and related models) available at https:// standards .iso .org/ iso
-iec/ 15938/ -18/ ed -2/ en (described in detail in 4.4) and follow the procedure described in 4.5.

© ISO/IEC 2025 – All rights reserved
4.2 Conformance testing for decoder
The decoder conformance is specified in ISO/IEC 15938-17:2024, Clause 7.
4.3 Conformance testing for bitstreams
The bitstream conformance is specified in ISO/IEC 15938-17:2024, Clause 7.
4.4 Models and reference bitstreams
A set of bitstreams and related neural network models is provided for conformance testing. When neural
network models are provided, they are the source models used to generate one or more compressed
bitstreams. The code defining these models is provided as part of the reference software distribution
available at https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -2/ en. Models and bitstreams are provided as
"reference_models_bitstreams_ed2_v1.1.zip".
A bitstream may be generated:
— from a commonly used neural network model,
— from synthetic data by encoding it with the reference encoder,
— by creating a conformant synthetic bitstream directly, without using an encoder.
Where applicable, the dataset used to train the model is mentioned for information only. However, the
dataset is not needed for conformance testing and thus not provided.
Table 1 summarizes the provided bitstreams for entire models, Table 2 summarizes the provided bitstreams
for incrementally coded models. The tables also list the features tested with the bitstreams, and the reference
encoder configuration used to generate the bitstream (where applicable). If decoding the bitstream requires
out-of-band parameters (e.g. information that is derived from the network topology description), those
parameters are also provided. We refer to source models and datasets using the following names:
— ImageNet for the dataset described in Reference [5],
— CIFAR-100 for the dataset described in Reference [6],
— DCase for the dataset described in Reference [7] and the model described in Reference [8],
— PASCAL VOC for the dataset described in Reference [12],
— chest_xray_v3 for the dataset described in Reference [13],
— MobileNet V2 for model described in Reference [9],
— UC12B for the model described in Reference [10],
— VGG16 for the model described in Reference [11], and
— ResNet18 for the model described in Reference [14].

© ISO/IEC 2025 – All rights reserved
Table 1 — Bitstreams and related models for conformance testing of coding entire models
Bitstream id Source Data-set Relevant tech- Features Reference encoder configuration
a
model nology tested
perf_map_ MobileNetV2 Image 6.3.4.3 sparsification qp_density = 2
sparse_ Net performance scan_order = 1
MobileNetV2. map approx_method = "codebook"
nctm qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_prune_ DCase DCase 6.3.4.3 pruning perfor- qp_density = 2
DCase.nctm mance map scan_order = 1
approx_method = "codebook"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_sparse_ UC12B CIFAR- 6.3.4.3 Sparsification qp_density = 2
prune_UC12B.nctm 100 and Pruning scan_order = 1
Performance approx_method = "codebook"
map qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_ VGG16 Image 6.3.4.3 Sparsification qp_density = 2
sparse_VGG16. Net Performance scan_order = 1
nctm Map (pruned approx_method = "codebook"
model) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
prune_tpl_cont_ DCase DCase 6.3.4.5 Prune Topol-
encode_tpl_only = True
sparse_bm_ ogy - sparse
partial_data_counter = 0
DCase.nctm bitmask
prune_tpl_cont_ VGG16 Image 6.3.4.5 Prune Topol-
encode_tpl_only = True
prune_bm_ Net ogy - prune
partial_data_counter = 0
VGG16.nctm bitmask
© ISO/IEC 2025 – All rights reserved
TTabablele 1 1 ((ccoonnttiinnueuedd))
Bitstream id Source Data-set Relevant tech- Features Reference encoder configuration
a
model nology tested
prune_tpl_cont_ VGG16 Image 6.3.4.5 Prune Topology
encode_tpl_only = True
comb_bm_ Net - combined
partial_data_counter = 0
VGG16.nctm bitmask
prune_tpl_cont_ DCase DCase 6.3.4.5 Prune Topology encode_tpl_only = True
prune_dictionary_ - prune diction- topology_indexed_reference_flag =
DCase.nctm ary False
partial_data_counter = 0
prune_tpl_cont_ ResNet50 Image 6.3.4.5 Prune Topology encode_tpl_only = True
prune_ Net - prune diction- topology_indexed_reference_flag =
dictionary_idx_ ary (indexed True
ResNet50.nctm elem id) partial_data_counter = 0
tpl_reflist_DCase. DCase DCase 6.3.4.5, 6.3.3.7 Topology Re- encode_tpl_only = True
nctm flist partial_data_counter = 0
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter
max_ndu_nnr_unit_size = 65536
ndu_size_65536.
nctm
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter
max_ndu_nnr_unit_size = 32768
ndu_size_32768.
nctm
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter
max_ndu_nnr_unit_size = 16384
ndu_size_16384.
nctm
partial_data_ DCase DCase 6.3.3.1 Partial data
counter_DCase_ counter
max_ndu_nnr_unit_size = 2048
ndu_size_2048.
nctm
partial_data_coun- DCase DCase 6.3.3.1 Partial data
ter_DCase_ndu_ counter max_ndu_nnr_unit_size = 1024
size_1024.nctm
deepCABAC_ ResNet50 Image 10 DeepCABAC en- see verify_all.sh
ResNet50_1_qp-38_ Net 9.2.1 / 9.3.1 tropy coding,
qp_density2.nctm uniform quan-
tization
dependent_ ResNet50 Image 9.2.3 / 9.3.3 Dependent see verify_all.sh
quantization_ Net 6.3.3.7 scalar quanti-
ResNet50_2_qp-38_ zation
qp_density2.nctm
deepCABAC_qp_ MobileNetV2 Image 9.3 QpDensity see verify_all.sh
density_Mobile- Net
NetV2_3_qp-38_
qp_density2.nctm
deepCABAC_ MobileNetV2 Image 9.3 QpDensity see verify_all.sh
qp_density_ Net
MobileNetV2_4_
qp-76_qp_density3.
nctm
block_scan_ ResNet50 Image 4.12 / 6.4.3.7 / Block scan see verify_all.sh
order_8x8_cabac_ Net 6.4.3.8 / 7.3.6 order / cabac
entry_points_ entry points
ResNet50_5_qp-38_
qp_density2.nctm
© ISO/IEC 2025 – All rights reserved
TTabablele 1 1 ((ccoonnttiinnueuedd))
Bitstream id Source Data-set Relevant tech- Features Reference encoder configuration
a
model nology tested
block_scan_or- ResNet50 Image 4.12 / 6.4.3.7 / Block scan see verify_all.sh
der_16x16_cabac_ Net 6.4.3.8 / 7.3.6 order / cabac
entry_points_ entry points
ResNet50_6_qp-38_
qp_density2.nctm
codebook_sig- MobileNetV2 Image 9.2.3 / 9.3.2 Code- see verify_all.sh
naling_Mo- Net book-based
bileNetV2_7_ quantization
qMse0.00001.nctm
local_scaling_ DCase DCase G.1.10 / 8.3.9 Local scaling see verify_all.sh
DCase_8_qp-38_qp_
density2.nctm
batchnorm_ MobileNetV2 Image 8.2.1 / 8.3.8 BatchNorm see verify_all.sh
folding_Mobile Net Folding
NetV2_9_qp-38_qp_
density2.nctm
out_of_band_ ResNet50 Image 6.3.3.7 / 6.4.3.7 Out-of-band see verify_all.sh
b
signaling_Res Net signaling
Net50_10_qp-38_
qp_density2.nctm
deepCABAC_8bit_ ResNet50 Image 9.2.1 / 9.3.1 Uniform quan- see verify_all.sh
ResNet50_PYT- Net tization with
zoo_11_qp0_qp_ limited preci-
density4.nctm sion (8bit)
deepCABAC_8bit_ MobileNet Image 9.2.1 / 9.3.1 Uniform quan- see verify_all.sh
MobileNetV2_PY- V2 Net tization with
Tzoo_12_qp0_qp_ limited preci-
density4.nctm sion (8bit)
deepCABAC_4bit_ VGG16 Image 9.2.1 / 9.3.1 Uniform quan- see verify_all.sh
VGG16_PYTzoo_13_ Net tization with
qp0_qp_density4. limited preci-
nctm sion (4bit)
deepCABAC_8bit_ UC12B CIFAR- 9.2.1 / 9.3.1 Uniform quan- see verify_all.sh
UC12B_14_qp0_qp_ 100 tization with
density4.nctm limited preci-
sion (8bit)
deepCABAC_4bit_ DCase DCase 9.2.1 / 9.3.1 Uniform quan- see verify_all.sh
DCase_15_qp0_qp_ tization with
density4.nctm limited preci-
sion (4bit)
perf_map_ MobileNetV2 Image 6.3.4.3 sparsification qp_density = 2
sparse_ Net performance scan_order = 1
MobileNetV2_bw8. map (8bit) approx_method = "uniform"
nctm qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
© ISO/IEC 2025 – All rights reserved
TTabablele 1 1 ((ccoonnttiinnueuedd))
Bitstream id Source Data-set Relevant tech- Features Reference encoder configuration
a
model nology tested
perf_map_prune_ DCase DCase 6.3.4.3 pruning per- qp_density = 2
DCase_bw8.nctm formance map scan_order = 1
(8bit) approx_method = "uniform"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_sparse_ UC12B CIFAR- 6.3.4.3 Sparsification qp_density = 2
prune_UC12B_bw8. 100 and Pruning scan_order = 1
nctm Performance approx_method = "uniform"
map (8bit) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
perf_map_ VGG16 Image 6.3.4.3 Sparsification qp_density = 2
sparse_VGG16_ Net Performance scan_order = 1
bw8. Map (pruned approx_method = "uniform"
nctm model) (8bit) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length_minus1 = 9
partial_data_counter = 0
partial_data_ DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size = 2048
counter_DCase_ counter (8bit)
ndu_size_2048_
approx_method = "uniform"
bw8.nctm
partial_data_coun- DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size = 1024
ter_DCase_ndu_ counter (8bit)
size_1024_bw8.
approx_method = "uniform"
nctm
a
Specifies the applicable subclause in ISO/IEC 15938-17:2024.
b
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
The bitstreams in Table 2 are provided for testing the decoding of a sequence of bitstreams representing
incremental updates of models. They shall be processed in the order indicated by the sequence number. The
character '#' in the name of the bitstream indicates the sequence number to be inserted, iterating over all
values in the range specified in column 'Seq.' (including the boundaries).
Bitstreams for the tested features Batchnorm folding (BNF), Row skipping (RS), Temporal context
modelling/adaptation (TCA) and History dependent significance probability (HDSP) use a common set of
parser arguments.
© ISO/IEC 2025 – All rights reserved
The parser arguments for technologies under test are 'bn_folding': True for enabling BNF, 'row_skip': True
for enabling RS, 'temporal_context': True for enabling TCA and 'hdsp': True for enabling HDSP. In a bitstream
set only one of these four parser arguments is True, namely that of the technology under test, all others are
set to False.
Bitstreams for the feature Predictive Residual Coding (PRE) use a common set of tools from the extended
compression tool set in conjunction with PRE: 'row_skip': True, 'temporal_context': True and 'hdsp': True.
Unless otherwise specified in the 'Reference encoder configuration' column of Table 2, all conformance
tests indicated by a 'Bitstream id' starting with 'UC10' share a common set of parser arguments: 'use_case':
'UC10', 'batch_size': 64. Likewise for those starting with 'UC14A', 'use_case': 'UC14A', 'qp': -48, 'batch_size':
100, 'model_arch': 'vgg16', 'unidirect_compress': False, 'train_cf': True are used. For PRE and SBT, different
qp values are reported as stated in the table, though the rest of the common parser arguments are the same
as above. 'UC10' and 'UC14A' refer to the use cases described in the evaluation framework for development
[15]
of NNC .
The common set of parser arguments during all BNF, RS, TCA and HDSP conformance tests is: 'disable_
finetuning': True, 'n_epochs': 4, 'approx_method': 'uniform', 'disable_dq': True, 'param_opt': True, 'qp_density':
2, 'qp_diff_offset': 0, 'opt_qp': False, 'lambda_scale': 0.0, 'cb_size_ratio': 5000, 'q_mse': 1e-05, 'nnr_pt_block_
enabled': 0, 'filter_sparsity': 0.0, 'filter_spars_gain': 1.0, 'lr': 1e-05, 'model_path': 'PYT_ZOO', 'ini_compress':
False, 'max_iterations': None, 'data_split_ratios': [1.0], 'iter_opt': False, 'qp_step': 1, 'bias': 0.125, 'lsa': 0.0, 'ioq':
False, 'oob_signaling': False, 'ce': 2, 'sbt_quantization_approach': 'combined', 'sbt_quantization_subapproach':
'random_round', 'export_model_diffs': False, 'sbt_bn_quant': True, 'sbt_mean_quant': False, 'pre': False,
'cache_len': 14, 'running_level': 'model'.
For PRE and SBT, the common set of parser arguments during all conformance tests are: 'disable_dq': True,
'param_opt': True, 'n_epochs': 4, 'qp_density': 2, 'qp_diff_offset': 0, 'lambda_scale': 0.0, 'cb_size_ratio': 5000,
'q_mse': 1e-05, 'nnr_pt_block_enabled': 0, ‘target_sparsity’: 0, 'filter_sparsity': 0.0, 'filter_spars_gain': 1.0,
'lr': 1e-05, 'model_path': 'PYT_ZOO', 'max_iterations': None, 'data_split_ratios': [1.0] , 'bias': 0.125, 'lsa': 0.0,
'ioq': False, 'oob_signaling': False , 'sbt_quantization_approach': 'combined', 'sbt_quantization_subapproach':
'random_round', 'sbt_bn_quant': False, 'sbt_mean_quant': False 'cache_len': 14, 'running_level': 'model',
'sparse': False, 'target_sparsity': 0.0.
All encoder-relevant parser arguments that are not listed above or differ from the mentioned ones are
specified in the “Reference encoder configuration” column of Table 2.
Table 2 — Bitstreams and related models for conformance testing of coding incremental models
Bitstream set Bitstream id Seq. Source Dataset Relevant Features Reference encoder
model tech-nol- tested configuration
a
ogy
UC10 MobileNet_v2 UC10_mobilenet_v2_ 0.3 MobileNet_ PAS- 8.3.8 Batchnorm 'model_arch': 'mo-
full model uniform_qp-34_BNF_ V2 CAL-VOC folding bilenet_v2', 'qp': -34,
epoch_#.nctm (BNF) ', 'bn_folding': True,
'sparse': False, 'target_
sparsity': 0.0
UC10 MobileNet_v2 UC10_mobilenet_ 0.3 MobileNet_ PAS- 8.3.8 Batchnorm 'model_arch': 'mo-
full model (with v2_uniform_qp-34_ V2 CAL-VOC folding bilenet_v2', 'qp': -34,
G.1.5
sparsification) BNF_sparsity_0.9_ (BNF) ', 'bn_folding': True,
G.1.6
epoch_#.nctm 'sparse': True, 'target_
sparsity': 0.9
UC10 ResNet18 full UC10_resnet18_uni- 0.3 ResNet18 PAS- 10.1.1.2 Row skip- 'model_arch':
model form_qp-37_RS_ CAL-VOC ping (RS) 'resnet18', 'qp': -37,
epoch_#.nctm ', 'row_skip': True,
'sparse': False, 'target_
sparsity': 0.0
UC10 ResNet18 full UC10_resnet18_uni- 0.3 ResNet18 PAS- 10.1.1.2 Row skip- 'model_arch':
model (with sparsifi- form_qp-37_sparsi- CAL-VOC ping (RS) 'resnet18', 'qp': -37,
G.1.5
cation) ty_0.9_RS_epoch_#. ', 'row_skip': True,
G.1.6
nctm 'sparse': True, 'target_
sparsity': 0.9
a
Specifies the applicable subclause in ISO/IEC 15938-17:2024.

© ISO/IEC 2025 – All rights reserved
TTabablele 2 2 ((ccoonnttiinnueuedd))
Bitstream set Bitstream id Seq. Source Dataset Relevant Features Reference encoder
model tech-nol- tested configuration
a
ogy
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.1.1.4 Temporal 'temporal_context':
sifier only center to form_qp-48_TCA_ xray_v3 context True, 'sparse': False,
clients classifier_only_ adaptation 'target_sparsity': 0.0
epoch_#_center.nctm (TCA)
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.1.1.4 Temporal 'temporal_context':
sifier only client 1 to form_qp-48_TCA_ xray_v3 context True, 'sparse': False,
center classifier_only_ adaptation 'target_sparsity': 0.0
epoch_#_client1. (TCA)
nctm
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.1.1.4 Temporal 'temporal_context':
sifier only client 2 to form_qp-48_TCA_ xray_v3 context True, 'sparse': False,
center classifier_only_ adaptation 'target_sparsity': 0.0
epoch_#_client2. (TCA)
nctm
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.1.1.4 Temporal 'temporal_context':
sifier only center to form_qp-48_TCA_ xray_v3 context True, 'sparse': True,
G.1.5
clients (with sparsifi- sparsity_0.8_classi- adaptation 'target_sparsity': 0.8
G.1.6
cation) fier_only_epoch_#_ (TCA)
center.nctm
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.1.1.4 Temporal 'temporal_context':
sifier only client 1 to form_qp-48_TCA_ xray_v3 context True, 'sparse': True,
G.1.5
center (with sparsifi- sparsity_0.8_classifi- adaptation 'target_sparsity': 0.8
G.1.6
cation) er_only_epoch_#_cli- (TCA)
ent1.nctm
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.1.1.4 Temporal 'temporal_context':
sifier only client 2 to form_qp-48_TCA_ xray_v3 context True, 'sparse': True,
G.1.5
center (with sparsifi- sparsity_0.8_classifi- adaptation 'target_sparsity': 0.8
G.1.6
cation) er_only_epoch_#_cli- (TCA)
ent2.nctm
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.2.1.4 / History 'hdsp': True, 'sparse':
sifier only center to form_qp-48_HDSP_ xray_v3 10.3.2.2 / dependent False, 'target_sparsi-
clients classifier_only_ 10.3.4.2.2 significance ty': 0.0
epoch_#_center.nctm probability
(HDSP)
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.2.1.4 / History 'hdsp': True, 'sparse':
sifier only client 1 to form_qp-48_HDSP_ xray_v3 10.3.2.2 / dependent False, 'target_sparsi-
center classifier_only_ 10.3.4.2.2 significance ty': 0.0
epoch_#_client1. probability
nctm (HDSP)
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.2.1.4 / History 'hdsp': True, 'sparse':
sifier only client 2 to form_qp-48_HDSP_ xray_v3 10.3.2.2 / dependent False, 'target_sparsi-
center classifier_only_ 10.3.4.2.2 significance ty': 0.0
epoch_#_client2. probability
nctm (HDSP)
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.2.1.4 / History 'hdsp': True, 'sparse':
sifier only center to form_qp-48_HDSP_ xray_v3 10.3.2.2 / dependent True, 'target_sparsi-
clients (with sparsifi- sparsity_0.8_classi- 10.3.4.2.2 significance ty': 0.8
cation) fier_only_epoch_#_ probability
G.1.5
center.nctm (HDSP)
G.1.6
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.2.1.4 / History 'hdsp': True, 'sparse':
sifier only client 1 to form_qp-48_HDSP_ xray_v3 10.3.2.2 / dependent True, 'target_sparsi-
center (with sparsifi- sparsity_0.8_classifi- 10.3.4.2.2 significance ty': 0.8
cation) er_only_epoch_#_cli- probability
G.1.5
ent1.nctm (HDSP)
G.1.6
a
Specifies the applicable subclause in ISO/IEC 15938-17:2024.

© ISO/IEC 2025 – All rights reserved
TTabablele 2 2 ((ccoonnttiinnueuedd))
Bitstream set Bitstream id Seq. Source Dataset Relevant Features Reference encoder
model tech-nol- tested configuration
a
ogy
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 10.2.1.4 / History 'hdsp': True, 'sparse':
sifier only client 2 to form_qp-48_HDSP_ xray_v3 10.3.2.2 / dependent True, 'target_sparsi-
center (with sparsifi- sparsity_0.8_classifi- 10.3.4.2.2 significance ty': 0.8
cation) er_only_epoch_#_cli- probability
G.1.5
ent2.nctm (HDSP)
G.1.6
UC10 ResNet18 clas- UC10_resnet18_uni- 0.3 ResNet18 PAS- 9.1.5 Stochastic ‘pre’: False, ‘approx_
sifier only form_sbt_qp-34_clas- CAL-VOC Binary Ter- method’: uniform_sbt,
sifier_only_epoch_#. nary (SBT) qp: -34, ‘train_cf’: True
nctm Quanti-za-
tion
UC10 VGG11 classifi- UC10_vgg11_uni- 0.3 VGG11 PAS- 9.1.5 Stochastic ‘pre’: False, ‘approx_
er only form_sbt_qp-36_clas- CAL-VOC Binary Ter- method’: uniform_sbt,
sifier_only_epoch_#. nary (SBT) qp: -36, ‘train_cf’: True
nctm Quanti-za-
tion
UC14A VGG16 full UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.5 Stochastic ‘pre’: False, ‘approx_
model center to form_sbt_qp-52_ xray_v3 Binary Ter- method’: uniform_sbt,
clients full_model_epoch_#_ nary (SBT) qp: -52, ‘train_cf’:
center.nctm Quanti-za- False
tion
UC14A VGG16 full UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.5 Stochastic ‘pre’: False, ‘approx_
model client 1 to form_sbt_qp-52_ xray_v3 Binary Ter- method’: uniform_sbt,
center full_model_epoch_#_ nary (SBT) qp: -52, ‘train_cf’:
client1.nctm Quanti-za- False
tion
UC14A VGG16 full UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.5 Stochastic ‘pre’: False, ‘approx_
model client 2 to form_sbt_qp-52_ xray_v3 Binary Ter- method’: uniform_sbt,
center full_model_epoch_#_ nary (SBT) qp: -52, ‘train_cf’:
client2.nctm Quanti-za- False
tion
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.6 Predictive ‘pre’: True, 'tempo-
sifier only client 1 to form_qp-48_PRE_ xray_v3 Residual ral_context': True,
center classifier_only_ Coding 'row_skip': True,
unidir_epoch_#_cli- (PRE) 'hdsp': True, ‘approx_
ent1.nctm method’: uniform,
'unidirect_compress':
True
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.6 Predictive ‘pre’: True, 'tempo-
sifier only client 2 to form_qp-48_PRE_ xray_v3 Residual ral_context': True,
center classifier_only_ Coding 'row_skip': True,
unidir_epoch_#_cli- (PRE) 'hdsp': True, ‘approx_
ent2.nctm method’: uniform,
'unidirect_compress':
True
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.6 Predictive ‘pre’: True, 'tempo-
sifier only center to form_qp-48_PRE_ xray_v3 Residual ral_context': True,
clients classifier_only_bidir_ Coding 'row_skip': True,
epoch_#_center.nctm (PRE) 'hdsp': True, ‘approx_
method’: uniform
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.6 Predictive ‘pre’: True, 'tempo-
sifier only client 1 to form_qp-48_PRE_ xray_v3 Residual ral_context': True,
center classifier_only_bidir_ Coding 'row_skip': True,
epoch_#_client1. (PRE) 'hdsp': True, ‘approx_
nctm method’: uniform
UC14A VGG16 clas- UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.6 Predictive ‘pre’: True, 'tempo-
sifier only client 2 to form_qp-48_PRE_ xray_v3 Residual ral_context': True,
center classifier_only_bidir_ Coding 'row_skip': True,
epoch_#_client2. (PRE) 'hdsp': True, ‘approx_
nctm method’: uniform
a
Specifies the applicable subclause in ISO/IEC 15938-17:2024.

© ISO/IEC 2025 – All rights reserved
TTabablele 2 2 ((ccoonnttiinnueuedd))
Bitstream set Bitstream id Seq. Source Dataset Relevant Features Reference encoder
model tech-nol- tested configuration
a
ogy
UC14A VGG16 full UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.6 Predictive ‘pre’: True, 'tempo-
model client 1 to form_qp-48_PRE_ xray_v3 Residual ral_context': True,
center full_model_unidir_ Coding 'row_skip': True,
epoch_#_client1. (PRE) 'hdsp': True, ‘approx_
nctm method’: uniform,
'unidirect_compress':
True
UC14A VGG16 full UC14A_vgg16_uni- 0.3 VGG16 chest_ 9.1.6 Predictive ‘pre’: True, 'tempo-
model client 2 to form_qp-48_PRE_ xray_v3 Residual ral_context': True,
center full_model_unidir_ Coding 'row_skip': True,
epoch_#_client2. (PRE) 'hdsp': True, ‘approx_
nctm method’: uniform,
'unidirect_compress':
True
a
Specifies the applicable subclause in ISO/IEC 15938-17:2024.
The bitstreams and models are provided at https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -2/ en/ reference
_models _bitstreams _ed2 _v1 .1 .zip.
The parameters of the bitstreams, for which no detailed parameters are listed, can be obtained from
the scripts given in column “Reference encoder configuration”, e.g. verify_all.sh in “run/conformance_
bitstreams” of the reference software. Scripts for generating all incremental bitstreams, as specified in
Table 2, and for decoder testing with these bitstreams are provided in the same directory (resp., “verify_
all_incremental.sh” and “decode_all_incremental.sh”). Furthermore, it also provides detailed reference
software configurations for each test. If out-of-band parameters are required, they are provided in a file
[16]
conforming to JSON (as specified in ISO/IEC 21778 ), specified in a footnote.
4.5 Procedure to test decoders
4.5.1 General
For testing a decoder, all bitstreams that cover mandatory features as specified in ISO/IEC 15938-17 shall be
tested. If support for optional features is claimed, also the bitstreams covering these features shall be tested.
On platforms with only integer or limited integer precision support, only those bitstreams indicated for the
supported bit depth ("8bit" or "4bit") shall be tested.
The procedure for testing features for decoding complete models and decoding a sequence of incremental
updates of a model follow the same principles, but differ in details. The two procedures are described in the
following subclauses.
4.5.2 Complete model
For each bitstream, the procedure includes the following steps:
a) Decode bitstream B with the reference decoder to obtain the decoded model R according to 4.5.4 if no
r
out-of-band parameters are specified and according to 4.5.5, otherwise.
b) Decode bitstream B with the decoder under test to obtain the decoded model R according to 4.5.4 if no
t
out-of-band parameters are specified and according to 4.5.5, otherwise.
c) Compare every tensor in R against the corresponding tensor in R . The test shall be considered as
r t
passed if the following conditions hold for every pair of tensors:
— If an integer tensor has been obtained as a result (decoding method specified in ISO/IEC 15938-17:2024,
7.3.2), the tensors shall be identical.

© ISO/IEC 2025 – All rights reserved
— If a floating point tensor has been obtained as the result of another decoding method, no corresponding
pair of floating point values shall have an absolute difference larger than 0,000001.
4.5.3 Incremental model
For each set of related bitstreams (as indicated in Table 2), the procedure includes the following steps:
a) Perform decoding of each of the bitstreams in the set with sequence numbers s (repeating this step for
each s from 0 to k, where k is the number of bitstreams in the set) in the specified order:
s s
1) Decode bitstream B with the reference decoder to obtain the decoded model R according to 4.5.4
r
if no out-of-band parameters are specified and according to 4.5.5, otherwise.
s s
2) Decode bitstream B with the decoder under test to obtain the decoded model R according to 4.5.4
t
if no out-of-band parameters are specified and according to 4.5.5, otherwise.
0 0 s s
3) If s equals 0, initialize the reconstructed models as R = R and R = R . Otherwise apply R and R
r r t t r t
as incremental updates to R and R , respectively (e.g. adding them to the tensor from the previous
r t
iteration).
b) Compare every tensor in R against the corresponding tensor in R . The test shall be considered as
r t
passed if the following conditions hold for every pair of tensors:
— If an integer tensor has been obtained as a result (decoding method specified in clause 7.3.2 of
ISO/IEC 15938-17), the tensors shall be identical.
— If a floating point tensor has been obtained as the result of another decoding method, no corresponding
pair of floating point values shall have an absolute difference larger than 0,000001.
4.5.4 Decoding self-contained NNC bitstreams
For self-contained bitstreams, the decoder shall not use any other inputs than the bitstream to be decoded.
4.5.5 Decoding NNC bitstreams using out-of-band parameters
The testing of decoding bitstreams using out-of-band parameters is agnostic to whether those are application
provided or decoded from a particular third party topology format. The decoder shall use only the input
bitstream and the parameters provided for this bitstream as specified in Table 1 as inputs.
4.6 Procedure to test bitstreams
In order to test bitstreams created with an encoder other than the reference encoder, the bitstream shall be
decoded using the reference decoder. The bitstream is considered conformant, if no errors occur during the
decoding process.
All bitstreams that have been created with the script “verify_all.sh” specified in Table 1 can be decoded with
the script “decode_all.sh” in “run/conformance_bitstreams” of the reference software. In the same directory,
the scripts “verify_all_incremental.sh” and “decode_all_incremental.sh” are provided for the bistreams in
Table 2. The bitstreams are considered conformant, if no errors occur during the decoding process.
5 Reference software
5.1 General
The purpose of this clause is to provide the following:
— Reference decoder software capable of decoding bitstreams that conform to ISO/IEC 15938-17 in a
manner that conforms to the decoding process specified in ISO/IEC 15938-17.
— Reference encoder software capable of producing bitstreams that conform to ISO/IEC 15938-17.

© ISO/IEC 2025 – All rights reserved
Some examples of uses that may be appropriate for the reference decoder software are as follows:
— As an illustration of how to perform the decoding process specified in ISO/IEC 15938-17.
— As the starting basis for the implementation of a decoder that conforms to ISO/IEC 15938-17.
— For testing the conformance of a decoder implementation with the decoding process specified in
ISO/IEC 15938-17.
— For testing the conformance of a bitstream to the constraints specified for bitstream conformance in
ISO/IEC 15938-17, as the software can detect and report many bitstream conformance violations.
However, the lack of the detection of any conformance violation by the reference decoder software should
not be considered as definitive proof that the bitstream conforms to all constraints specified for bitstream
conformance in ISO/IEC 15938-17.
Some examples of uses that may be appropriate for the refer
...