ISO/IEC 23094-4:2022
(Amendment)Information technology — General video coding — Part 4: Conformance and reference software for essential video coding
Information technology — General video coding — Part 4: Conformance and reference software for essential video coding
This document specifies a set of tests and procedures designed to indicate whether encoders or decoders essential video coding (EVC), which contains tests and a reference software designed to verify whether bitstreams and decoders meet normative requirements specified in ISO/IEC 23094-1. An encoder can claim conformance to ISO/IEC 23094-1 if the bitstreams that it generates are conforming bitstreams. Characteristics of coded bitstreams and decoders are defined in ISO/IEC 23094-1. Decoder characteristics define the properties and capabilities of the applied decoding process. The capabilities of a decoder specify which bitstreams the decoder can decode and reconstruct. A bitstream can be decoded by a decoder if the characteristics of the bitstream are within the specified decoder capabilities.
Technologies de l'information — Codage vidéo général — Partie 4: Conformité et logiciel de référence pour le codage vidéo essentiel
General Information
Buy Standard
Standards Content (sample)
INTERNATIONAL ISO/IEC
STANDARD 23094-4
First edition
2022-01
Information technology — General
video coding —
Part 4:
Conformance and reference software
for essential video coding
Technologies de l'information — Codage vidéo général —
Partie 4: Conformité et logiciel de référence pour le codage vidéo
essentiel
Reference number
ISO/IEC 23094-4:2022(E)
© ISO/IEC 2022
---------------------- Page: 1 ----------------------
ISO/IEC 23094-4:2022(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2022
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 23094-4:2022(E)
Contents Page
Foreword ........................................................................................................................................................................................................................................iv
1 Scope ................................................................................................................................................................................................................................. 1
2 Normative references ..................................................................................................................................................................................... 1
3 Terms and definitions .................................................................................................................................................................................... 1
4 Abbreviated terms ............................................................................................................................................................................................. 2
5 Conventions ............................................................................................................................................................................................................... 2
6 Conformance testing for ISO/IEC 23094-1 ................................................................................................................................ 2
6.1 Introduction .............................................................................................................................................................................................. 2
6.2 Bitstream conformance .................................................................................................................................................................. 2
6.3 Decoder conformance ...................................................................................................................................................................... 2
6.4 Procedure to test bitstreams ..................................................................................................................................................... 2
6.5 Procedure to test decoder conformance ......................................................................................................................... 3
6.5.1 Conformance bitstreams ............................................................................................................................................ 3
6.5.2 Contents of the bitstream file ................................................................................................................................. 3
6.5.3 Requirements on output of the decoding process and timing ................................................... 3
6.5.4 Recommendations............................................................................................................................................................. 4
6.5.5 Static tests for output order conformance .................................................................................................. 4
6.5.6 Dynamic tests for output timing conformance ....................................................................................... 4
6.5.7 Decoder conformance test of a particular profile and level ........................................................ 5
6.6 Specification of the test bitstreams ..................................................................................................................................... 5
6.6.1 General ........................................................................................................................................................................................ 5
6.6.2 Test bitstreams .................................................................................................................................................................... 6
6.7 Test suites for ISO/IEC 23094-1 ...........................................................................................................................................25
7 Reference software description .......................................................................................................................................................30
7.1 General ........................................................................................................................................................................................................30
7.2 ETM repository ................................................................................................................................................................................... 31
7.3 Encoder and decoder usage ..................................................................................................................................................... 31
iii© ISO/IEC 2022 – All rights reserved
---------------------- Page: 3 ----------------------
ISO/IEC 23094-4:2022(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.The procedures used to develop this document and those intended for its further maintenance
are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria
needed for the different types of document should be noted. This document was drafted in
accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or
www.iec.ch/members_experts/refdocs).Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC
list of patent declarations received (see https://patents.iec.ch).Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see
www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 23094 series can be found on the ISO website and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.© ISO/IEC 2022 – All rights reserved
---------------------- Page: 4 ----------------------
INTERNATIONAL STANDARD ISO/IEC 23094-4:2022(E)
Information technology — General video coding —
Part 4:
Conformance and reference software for essential video
coding
1 Scope
This document specifies a set of tests and procedures designed to indicate whether encoders or
decoders essential video coding (EVC), which contains tests and a reference software designed to verify
whether bitstreams and decoders meet normative requirements specified in ISO/IEC 23094-1. An
encoder can claim conformance to ISO/IEC 23094-1 if the bitstreams that it generates are conforming
bitstreams. Characteristics of coded bitstreams and decoders are defined in ISO/IEC 23094-1. Decoder
characteristics define the properties and capabilities of the applied decoding process. The capabilities
of a decoder specify which bitstreams the decoder can decode and reconstruct. A bitstream can be
decoded by a decoder if the characteristics of the bitstream are within the specified decoder capabilities.
2 Normative referencesThe following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 23094-1:2020, Information technology — General video coding — Part 1: Essential video coding
ISO/IEC 9899, Information technology — Programming languages — CISO/IEC/IEEE 9945, Information technology — Portable Operating System Interface (POSIX®) Base
Specifications, Issue 73 Terms and definitions
For the purposes of this document, the terms and definitions and symbols specified in ISO/IEC 23094-1
and the following apply.ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp— IEC Electropedia: available at https:// www .electropedia .org/
3.1
bitstream
sequence of bits, in the form of a NAL unit stream or a raw bitstream, that forms the representation of
coded pictures and associated data forming one or more coded video sequences3.2
decoder
embodiment of a process that operates on a bitstream and may conform to the decoding process
requirements specified for conformanceNote 1 to entry: The decoder does not include the display process, which is outside the scope of this document
© ISO/IEC 2022 – All rights reserved---------------------- Page: 5 ----------------------
ISO/IEC 23094-4:2022(E)
3.3
encoder
embodiment of a process that produces a bitstream
Note 1 to entry: The process is not specified in this document (except in regard to identification of the reference
software encoder).3.4
reference software decoder
software which may decode a bitstream (3.1) encoded according to the syntax structure
4 Abbreviated termsETM test model of essential video coding
HRD hypothetical reference decoder
5 Conventions
For the purposes of this document, relevant conventions are specified in Clause 5 of
ISO/IEC 23094-1:2020.6 Conformance testing for ISO/IEC 23094-1
6.1 Introduction
The following subclauses specify the tests for verifying conformance of video bitstreams as well as
decoders. These tests make use of test data (bitstream test suites) provided as an electronic annex to
this document and the reference software decoder specified in ISO/IEC 23094-1.The electronic annex to this document is available at the following address:
― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/
6.2 Bitstream conformance
Bitstream conformance is specified by Clause C.4 of ISO/IEC 23094-1:2020.
6.3 Decoder conformance
Decoder conformance is specified by Clause C.5 of ISO/IEC 23094-1:2020.
6.4 Procedure to test bitstreams
A bitstream that claims conformance with ISO/IEC 23094-1 should pass the following normative test.
The bitstream should be decoded by processing it with the reference software decoder. When processed
by the reference software decoder, the bitstream should not cause any error or non-conformance
messages to be reported by the reference software decoder. This test should not be applied to bitstreams
that are known to contain errors introduced by transmission, as such errors are highly likely to result
in bitstreams that lack conformance to ISO/IEC 23094-1.Successfully passing the reference software decoder test provides only a strong presumption that the
bitstream under test is conforming to the video layer, i.e., that it does indeed meet all the requirements
for the video layer (except Annexes C, D and E) specified in ISO/IEC 23094-1:2020 that are tested by the
reference software decoder.© ISO/IEC 2022 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC 23094-4:2022(E)
Additional tests may be necessary to more thoroughly check that the bitstream properly meets all the
requirements specified in ISO/IEC 23094-1:2020 including the hypothetical reference decoder (HRD)
conformance (based on Annexes C, D and E). These complementary tests may be performed using other
video bitstream verifiers that perform more complete tests than those implemented by the reference
software decoder.When testing a bitstream for conformance, it may also be useful to test whether or not the bitstream
follows the informative recommendations specified in ISO/IEC 23094-1.To check correctness of a bitstream, it is necessary to parse the entire bitstream and to extract all the
syntax elements and other values derived from those syntactic elements and used by the decoding
process specified in ISO/IEC 23094-1.A verifier may not necessarily perform all stages of the decoding process specified in ISO/IEC 23094-1
in order to verify bitstream correctness. Many tests can be performed on syntax elements in a state
prior to their use in some processing stages.6.5 Procedure to test decoder conformance
6.5.1 Conformance bitstreams
A bitstream has values of profile_idc and level_idc corresponding to a set of specified constraints on
a bitstream for which a decoder conforming to a specified profile and level is required in Annex A of
ISO/IEC 23094-1:2020 to properly perform the decoding process.6.5.2 Contents of the bitstream file
The conformance bitstreams are included in this document as an electronic attachment available at the
following address:― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/
The following information is included in a single zipped file for each such bitstream.
― bitstream, and― decoded pictures or hashes of decoded pictures (may not be present), and
― short description of the bitstream, and
― trace file (results while decoding the bitstream, in ASCII format).
In cases where the decoded pictures or hashes of decoded pictures are not available, the reference
software decoder should be used to generate the necessary reference decoded pictures from the
bitstream.6.5.3 Requirements on output of the decoding process and timing
Two classes of decoder conformance are specified:
― output order conformance, and
― output timing conformance.
The output of the decoding process is specified in Clause 8 and Annex C of ISO/IEC 23094-1:2020.
For output order conformance, it is a requirement that all of the decoded pictures specified for output
in Annex C of ISO/IEC 23094-1:2020 should be output by a conforming decoder in the specified order
and that the values of the decoded samples in all of the pictures that are output should be (exactly equal
to) the values specified in Clause 8 of ISO/IEC 23094-1:2020.© ISO/IEC 2022 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC 23094-4:2022(E)
For output timing conformance, it is a requirement that a conforming decoder should also output the
decoded samples at the rates and times specified in Annex C of ISO/IEC 23094-1:2020.
The display process, which ordinarily follows the output of the decoding process, is outside the scope of
this International Standard.6.5.4 Recommendations
In addition to the requirements, it is desirable that conforming decoders implement various
informative recommendations specified in ISO/IEC 23094-1. This subclause discusses some of these
recommendations.It is recommended that a conforming decoder be able to resume the decoding process as soon as possible
after the loss or corruption of part of a bitstream. In most cases it is possible to resume decoding at the
next slice header. It is recommended that a conforming decoder be able to perform concealment for the
coding tree blocks or video packets for which all the coded data has not been received.
6.5.5 Static tests for output order conformanceStatic tests of a video decoder require testing of the decoded samples. This clause will explain how
this test can be accomplished when the decoded samples at the output of the decoding process are
available. It may not be possible to perform this type of test with a production decoder (due to the
lack of an appropriate accessible interface in the design at which to perform the test). In that case this
test should be performed by the manufacturer during the design and development phase. Static tests
are used for testing the decoding process. The test will check that the values of the samples decoded
by the decoder under test should be identical to the values of the samples decoded by the reference
decoder. When a hash of the values of the samples of the decoded pictures is attached to the bitstream
file, a corresponding hash operation performed on the values of the samples of the decoded pictures
produced by the decoder under test should produce the same results.6.5.6 Dynamic tests for output timing conformance
Dynamic tests are applied to check that all the decoded samples are output and that the timing of
the output of the decoder's decoded samples conforms to the specification of Clause 8 and Annex C of
ISO/IEC 23094-1:2020, and to verify that the HRD models (as specified by the CPB and DPB specification
in Annex C of ISO/IEC 23094-1:2020) are not violated when the bits of the bitstream are delivered at the
proper rate.The dynamic test is often easier to perform on a complete decoding system, which may include a
systems decoder, a video decoder and a display process. It may be possible to record the output of the
display process and to check that display order and timing of decoded pictures are correct at the output
of the display process. However, since the display process is not within the scope of ISO/IEC 23094-1,
there may be cases where the output of the display process differs in timing or value even though the
video decoder is conforming. In this case, the output of the video decoder itself (before the display
process) would need to be captured in order to perform the dynamic tests on the video decoder. In
particular the output order and timing of the decoded pictures should be correct.
If buffering period and picture timing SEI messages are included in the test bitstream, HRD conformance
should be verified using the values of initial_cpb_removal_delay, initial_cpb_removal_delay_offset, cpb_
removal_delay and dpb_removal_delay that are included in the bitstream.If buffering period and picture timing SEI messages are not included in the bitstream, the following
inferences should be made to generate the missing parameters:― fixed_pic_rate_flag should be inferred to be equal to 1, and
― low_delay_hrd_flag should be inferred to be equal to 0, and
― cbr_flag should be inferred to be equal to 0, and
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 23094-4:2022(E)
― The frame rate of the bitstream should be inferred to be equal to the frame rate value specified in
the corresponding Table 1 of 6.7, where the bitstream is listed. If this is missing, then a frame rate of
either 25 or 30 000 ÷ 1 001 can be inferred, and― time_scale should be set equal to 90 000 and the value of num_units_in_tick should be computed
based on frame rate, and― The bit rate of the bitstream should be inferred to be equal to the maximum value for the level
specified in Table A.1 in ISO/IEC 23094-1:2020, and― CPB and DPB sizes should be inferred to be equal to the maximum value for the level specified in
Table A.1 in ISO/IEC 23094-1:2020.With the above inferences, the HRD should be operated as follows:
― The CPB is filled starting at time t = 0, until it is full, before removal of the first access unit. This
means that the initial_cpb_removal_delay should be inferred to be equal to the total CPB buffer size
divided by the bit rate divided by 90 000 (rounded downwards) and initial_cpb_removal_delay_
offset should be inferred to be equal to zero, and― The first access unit is removed at time t = initial_cpb_removal_delay ÷ 90 000 and subsequent
access units are removed at intervals based on the frame distance, i.e., 2 * (90 000 ÷ num_units_in_
tick) or the field distance, i.e., (90 000 / num_units_in_tick), depending on whether the access unit
is coded as a frame picture or field picture, and― Using these inferences, the CPB will not overflow or underflow and the DPB will not overflow.
6.5.7 Decoder conformance test of a particular profile and levelIn order for a decoder of a particular profile and level to claim output order conformance to
ISO/IEC 23094-1 as specified by this document, the decoder should successfully pass the static test
specified in subclause 6.5.5 with all the bitstreams of the normative test suite specified for testing
decoders of this particular profile and level combination.In order for a decoder of a particular profile and level to claim output timing conformance to
ISO/IEC 23094-1 as specified by this document, the decoder should successfully pass both the static test
specified in subclause 6.5.5 and the dynamic test specified in subclause 6.5.6 with all the bitstreams of
the normative test suite specified for testing decoders of this particular profile and level. Table 1 specify
the normative test suites for each profile and level combination. The test suite for a particular profile
and level combination is the list of bitstreams that are marked with an 'X' in the column corresponding
to that profile and level combination.'X' indicates that the bitstream is designed to test both the dynamic and static conformance of the
decoder.The bitstream column specifies the bitstream used for each test.
A decoder that conforms to the Baseline profile or Main profile with 10 bit-depth at a specific level
should be capable of decoding the specified bitstreams in Table 1.6.6 Specification of the test bitstreams
6.6.1 General
Some characteristics of each bitstream listed in Table 1 are specified in this clause. In Table 1, the
value "29,97" should be interpreted as an approximation of an exact value of 30 000 ÷ 1 001 and the
value "59,94" should be interpreted as an approximation of an exact value of 60 000 ÷ 1 001.
A set of test vectors have been provided for conformance and are available at the following address:
― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/© ISO/IEC 2022 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 23094-4:2022(E)
6.6.2 Test bitstreams
6.6.2.1 Test bitstream BP_SET_A
Specification: Streams with sets of coding tools in Baseline profile.
Functional stage: Test the decoding process of Baseline profile.
Purpose: Check that the decoder can properly decode bitstreams in which the full set of coding tools in
Baseline profile is enabled.6.6.2.2 Test bitstream BP_SET_B
Specification: Streams with sets of coding tools in Baseline Still Picture profile.
Functional stage: Test the decoding process of Baseline Still Picture profile.Purpose: Check that the decoder can properly decode bitstreams in which the full set of coding tools in
Baseline Still Picture profile is enabled.6.6.2.3 Test bitstream MP_MIN_A
Specification: Streams with all tools disabled in Main profile in which SPS tool flags are equal to 0 in
Main profile.Functional stage: Test the decoding process of Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools
are disabled in Main profile.6.6.2.4 Test bitstream MP_MIN_B
Specification: Streams with all tools disabled in Main Still Picture profile in which SPS tool flags are
equal to 0.Functional stage: Test the decoding process of Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools
are disabled in Main Still Picture profile6.6.2.5 Test bitstream MP_SET_A
Specification: Streams with all tools enabled in Main profile in which SPS tool flags are equal to 1 in
Main profile.Functional stage: Test the decoding process of Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools
are enabled in Main profile.6.6.2.6 Test bitstream MP_SET_B
Specification: Streams with all tools enabled in Main Still Picture profile in which SPS tool flags are
equal to 1 in Main profile.Functional stage: Test the decoding process of Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools
are enabled in Main Still Picture profile.© ISO/IEC 2022 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC 23094-4:2022(E)
6.6.2.7 Test bitstream CTU_A
Specification: Streams with max CTU size of 64 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 64 is enabled
in Main profile.6.6.2.8 Test bitstream CTU_B
Specification: Streams with max CTU size of 32 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 32 is enabled
in Main profile.6.6.2.9 Test bitstream CTU_C
Specification: Streams with max CTU size of 128 and min CTU size of 8 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 128 and min
CTU size of 8 are enabled in Main profile.6.6.2.10 Test bitstream CTU_D
Specification: Streams with CTU size of 32 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which CTU size of 32 is enabled in
Main profile.6.6.2.11 Test bitstream CTU_E
Specification: Streams with max CTU size of 64 enabled in Main Still Picture profile.
Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 64 is enabled
in Main Still Picture profile.6.6.2.12 Test bitstream CTU_F
Specification: Streams with max CTU size of 32 enabled in Main Still Picture profile.
Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 32 is enabled
in Main Still Picture profile.6.6.2.13 Test bitstream CTU_G
Specification: Streams with max CTU size of 128 and min CTU size of 8 enabled in Main Still Picture
profile.Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.
© ISO/IEC 2022 – All rights reserved---------------------- Page: 11 ----------------------
ISO/IEC 23094-4:2022(E)
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 128 and min
CTU size of 8 are enabled in Main Still Picture profile.6.6.2.14 Test bitstream CTU_H
Specification: Streams with CTU size of 32 enabled in Main Still Picture profile.
Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which CTU size of 32 is enabled in
Main Still Picture profile.6.6.2.15 Test bitstream BTT_A
Specification: Streams with BTT tool disabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which BTT tool is disabled in Main
profile.6.6.2.16 Test bitstream BTT_B
Specification: Streams with only BTT tool enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which only BTT tool is enabled in
Main profile....
FINAL
INTERNATIONAL ISO/IEC
DRAFT
STANDARD FDIS
23094-4
ISO/IEC JTC 1/SC 29
Information technology — General
Secretariat: JISC
video coding —
Voting begins on:
2021-10-05
Part 4:
Voting terminates on:
Conformance and reference software
2021-11-30
for essential video coding
RECIPIENTS OF THIS DRAFT ARE INVITED TO
SUBMIT, WITH THEIR COMMENTS, NOTIFICATION
OF ANY RELEVANT PATENT RIGHTS OF WHICH
THEY ARE AWARE AND TO PROVIDE SUPPOR TING
DOCUMENTATION.
IN ADDITION TO THEIR EVALUATION AS
Reference number
BEING ACCEPTABLE FOR INDUSTRIAL, TECHNO-
ISO/IEC FDIS 23094-4:2021(E)
LOGICAL, COMMERCIAL AND USER PURPOSES,
DRAFT INTERNATIONAL STANDARDS MAY ON
OCCASION HAVE TO BE CONSIDERED IN THE
LIGHT OF THEIR POTENTIAL TO BECOME STAN-
DARDS TO WHICH REFERENCE MAY BE MADE IN
NATIONAL REGULATIONS. © ISO/IEC 2021
---------------------- Page: 1 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
Contents Page
Foreword ........................................................................................................................................................................................................................................iv
1 Scope ................................................................................................................................................................................................................................. 1
2 Normative references ..................................................................................................................................................................................... 1
3 Terms and definitions .................................................................................................................................................................................... 1
4 Abbreviated terms ............................................................................................................................................................................................. 2
5 Conventions ............................................................................................................................................................................................................... 2
6 Conformance testing for ISO/IEC 23094-1 ................................................................................................................................ 2
6.1 Introduction .............................................................................................................................................................................................. 2
6.2 Bitstream conformance .................................................................................................................................................................. 2
6.3 Decoder conformance ...................................................................................................................................................................... 2
6.4 Procedure to test bitstreams ..................................................................................................................................................... 2
6.5 Procedure to test decoder conformance ......................................................................................................................... 3
6.5.1 Conformance bitstreams ............................................................................................................................................ 3
6.5.2 Contents of the bitstream file ................................................................................................................................. 3
6.5.3 Requirements on output of the decoding process and timing ................................................... 3
6.5.4 Recommendations............................................................................................................................................................. 4
6.5.5 Static tests for output order conformance .................................................................................................. 4
6.5.6 Dynamic tests for output timing conformance ....................................................................................... 4
6.5.7 Decoder conformance test of a particular profile and level ........................................................ 5
6.6 Specification of the test bitstreams ..................................................................................................................................... 5
6.6.1 General ........................................................................................................................................................................................ 5
6.6.2 Test bitstreams .................................................................................................................................................................... 6
6.7 Test suites for ISO/IEC 23094-1 ...........................................................................................................................................25
7 Reference software description .......................................................................................................................................................30
7.1 General ........................................................................................................................................................................................................30
7.2 ETM repository ................................................................................................................................................................................... 31
7.3 Encoder and decoder usage ..................................................................................................................................................... 31
iii© ISO/IEC 2021 – All rights reserved
---------------------- Page: 3 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.The procedures used to develop this document and those intended for its further maintenance
are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria
needed for the different types of document should be noted. This document was drafted in
accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or
www.iec.ch/members_experts/refdocs).Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC
list of patent declarations received (see https://patents.iec.ch).Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see
www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 23094 series can be found on the ISO website and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.© ISO/IEC 2021 – All rights reserved
---------------------- Page: 4 ----------------------
FINAL DRAFT INTERNATIONAL STANDARD ISO/IEC FDIS 23094-4:2021(E)
Information technology — General video coding —
Part 4:
Conformance and reference software for essential video
coding
1 Scope
This document specifies a set of tests and procedures designed to indicate whether encoders or
decoders essential video coding (EVC), which contains tests and a reference software designed to verify
whether bitstreams and decoders meet normative requirements specified in ISO/IEC 23094-1. An
encoder can claim conformance to ISO/IEC 23094-1 if the bitstreams that it generates are conforming
bitstreams. Characteristics of coded bitstreams and decoders are defined in ISO/IEC 23094-1. Decoder
characteristics define the properties and capabilities of the applied decoding process. The capabilities
of a decoder specify which bitstreams the decoder can decode and reconstruct. A bitstream can be
decoded by a decoder if the characteristics of the bitstream are within the specified decoder capabilities.
2 Normative referencesThe following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 23094-1:2020, Information technology — General video coding — Part 1: Essential video coding
ISO/IEC 9899, Information technology — Programming languages — CISO/IEC/IEEE 9945, Information technology — Portable Operating System Interface (POSIX®) Base
Specifications, Issue 73 Terms and definitions
For the purposes of this document, the terms and definitions and symbols specified in ISO/IEC 23094-1
and the following apply.ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp— IEC Electropedia: available at https:// www .electropedia .org/
3.1
bitstream
sequence of bits, in the form of a NAL unit stream or a raw bitstream, that forms the representation of
coded pictures and associated data forming one or more coded video sequences3.2
decoder
embodiment of a process that operates on a bitstream and may conform to the decoding process
requirements specified for conformanceNote 1 to entry: The decoder does not include the display process, which is outside the scope of this document
© ISO/IEC 2021 – All rights reserved---------------------- Page: 5 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
3.3
encoder
embodiment of a process that produces a bitstream
Note 1 to entry: The process is not specified in this document (except in regard to identification of the reference
software encoder).3.4
reference software decoder
software which may decode a bitstream (3.1) encoded according to the syntax structure
4 Abbreviated termsETM test model of essential video coding
HRD hypothetical reference decoder
5 Conventions
For the purposes of this document, relevant conventions are specified in Clause 5 of ISO/IEC 23094-
1:2020.6 Conformance testing for ISO/IEC 23094-1
6.1 Introduction
The following subclauses specify the tests for verifying conformance of video bitstreams as well as
decoders. These tests make use of test data (bitstream test suites) provided as an electronic annex to
this document and the reference software decoder specified in ISO/IEC 23094-1.The electronic annex to this document is available at the following address:
― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/
6.2 Bitstream conformance
Bitstream conformance is specified by Clause C.4 of ISO/IEC 23094-1:2020.
6.3 Decoder conformance
Decoder conformance is specified by Clause C.5 of ISO/IEC 23094-1:2020.
6.4 Procedure to test bitstreams
A bitstream that claims conformance with ISO/IEC 23094-1 should pass the following normative test.
The bitstream should be decoded by processing it with the reference software decoder. When processed
by the reference software decoder, the bitstream should not cause any error or non-conformance
messages to be reported by the reference software decoder. This test should not be applied to bitstreams
that are known to contain errors introduced by transmission, as such errors are highly likely to result
in bitstreams that lack conformance to ISO/IEC 23094-1.Successfully passing the reference software decoder test provides only a strong presumption that the
bitstream under test is conforming to the video layer, i.e., that it does indeed meet all the requirements
for the video layer (except Annexes C, D and E) specified in ISO/IEC 23094-1:2020 that are tested by the
reference software decoder.© ISO/IEC 2021 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
Additional tests may be necessary to more thoroughly check that the bitstream properly meets all the
requirements specified in ISO/IEC 23094-1:2020 including the hypothetical reference decoder (HRD)
conformance (based on Annexes C, D and E). These complementary tests may be performed using other
video bitstream verifiers that perform more complete tests than those implemented by the reference
software decoder.When testing a bitstream for conformance, it may also be useful to test whether or not the bitstream
follows the informative recommendations specified in ISO/IEC 23094-1.To check correctness of a bitstream, it is necessary to parse the entire bitstream and to extract all the
syntax elements and other values derived from those syntactic elements and used by the decoding
process specified in ISO/IEC 23094-1.A verifier may not necessarily perform all stages of the decoding process specified in ISO/IEC 23094-1
in order to verify bitstream correctness. Many tests can be performed on syntax elements in a state
prior to their use in some processing stages.6.5 Procedure to test decoder conformance
6.5.1 Conformance bitstreams
A bitstream has values of profile_idc and level_idc corresponding to a set of specified constraints on a
bitstream for which a decoder conforming to a specified profile and level is required in Annex A of ISO/
IEC 23094-1:2020 to properly perform the decoding process.6.5.2 Contents of the bitstream file
The conformance bitstreams are included in this document as an electronic attachment available at the
following address:― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/
The following information is included in a single zipped file for each such bitstream.
― bitstream, and― decoded pictures or hashes of decoded pictures (may not be present), and
― short description of the bitstream, and
― trace file (results while decoding the bitstream, in ASCII format).
In cases where the decoded pictures or hashes of decoded pictures are not available, the reference
software decoder should be used to generate the necessary reference decoded pictures from the
bitstream.6.5.3 Requirements on output of the decoding process and timing
Two classes of decoder conformance are specified:
― output order conformance, and
― output timing conformance.
The output of the decoding process is specified in Clause 8 and Annex C of ISO/IEC 23094-1:2020.
For output order conformance, it is a requirement that all of the decoded pictures specified for output
in Annex C of ISO/IEC 23094-1:2020 should be output by a conforming decoder in the specified order
and that the values of the decoded samples in all of the pictures that are output should be (exactly equal
to) the values specified in Clause 8 of ISO/IEC 23094-1:2020.© ISO/IEC 2021 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
For output timing conformance, it is a requirement that a conforming decoder should also output the
decoded samples at the rates and times specified in Annex C of ISO/IEC 23094-1:2020.
The display process, which ordinarily follows the output of the decoding process, is outside the scope of
this International Standard.6.5.4 Recommendations
In addition to the requirements, it is desirable that conforming decoders implement various
informative recommendations specified in ISO/IEC 23094-1. This subclause discusses some of these
recommendations.It is recommended that a conforming decoder be able to resume the decoding process as soon as possible
after the loss or corruption of part of a bitstream. In most cases it is possible to resume decoding at the
next slice header. It is recommended that a conforming decoder be able to perform concealment for the
coding tree blocks or video packets for which all the coded data has not been received.
6.5.5 Static tests for output order conformanceStatic tests of a video decoder require testing of the decoded samples. This clause will explain how
this test can be accomplished when the decoded samples at the output of the decoding process are
available. It may not be possible to perform this type of test with a production decoder (due to the
lack of an appropriate accessible interface in the design at which to perform the test). In that case this
test should be performed by the manufacturer during the design and development phase. Static tests
are used for testing the decoding process. The test will check that the values of the samples decoded
by the decoder under test should be identical to the values of the samples decoded by the reference
decoder. When a hash of the values of the samples of the decoded pictures is attached to the bitstream
file, a corresponding hash operation performed on the values of the samples of the decoded pictures
produced by the decoder under test should produce the same results.6.5.6 Dynamic tests for output timing conformance
Dynamic tests are applied to check that all the decoded samples are output and that the timing of the
output of the decoder's decoded samples conforms to the specification of Clause 8 and Annex C of ISO/
IEC 23094-1:2020, and to verify that the HRD models (as specified by the CPB and DPB specification in
Annex C of ISO/IEC 23094-1:2020) are not violated when the bits of the bitstream are delivered at the
proper rate.The dynamic test is often easier to perform on a complete decoding system, which may include a
systems decoder, a video decoder and a display process. It may be possible to record the output of the
display process and to check that display order and timing of decoded pictures are correct at the output
of the display process. However, since the display process is not within the scope of ISO/IEC 23094-1,
there may be cases where the output of the display process differs in timing or value even though the
video decoder is conforming. In this case, the output of the video decoder itself (before the display
process) would need to be captured in order to perform the dynamic tests on the video decoder. In
particular the output order and timing of the decoded pictures should be correct.
If buffering period and picture timing SEI messages are included in the test bitstream, HRD conformance
should be verified using the values of initial_cpb_removal_delay, initial_cpb_removal_delay_offset, cpb_
removal_delay and dpb_removal_delay that are included in the bitstream.If buffering period and picture timing SEI messages are not included in the bitstream, the following
inferences should be made to generate the missing parameters:― fixed_pic_rate_flag should be inferred to be equal to 1, and
― low_delay_hrd_flag should be inferred to be equal to 0, and
― cbr_flag should be inferred to be equal to 0, and
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
― The frame rate of the bitstream should be inferred to be equal to the frame rate value specified in
the corresponding Table 1 of 6.7, where the bitstream is listed. If this is missing, then a frame rate of
either 25 or 30 000 ÷ 1 001 can be inferred, and― time_scale should be set equal to 90 000 and the value of num_units_in_tick should be computed
based on frame rate, and― The bit rate of the bitstream should be inferred to be equal to the maximum value for the level
specified in Table A.1 in ISO/IEC 23094-1:2020, and― CPB and DPB sizes should be inferred to be equal to the maximum value for the level specified in
Table A.1 in ISO/IEC 23094-1:2020.With the above inferences, the HRD should be operated as follows:
― The CPB is filled starting at time t = 0, until it is full, before removal of the first access unit. This
means that the initial_cpb_removal_delay should be inferred to be equal to the total CPB buffer size
divided by the bit rate divided by 90 000 (rounded downwards) and initial_cpb_removal_delay_
offset should be inferred to be equal to zero, and― The first access unit is removed at time t = initial_cpb_removal_delay ÷ 90 000 and subsequent
access units are removed at intervals based on the frame distance, i.e., 2 * (90 000 ÷ num_units_in_
tick) or the field distance, i.e., (90 000 / num_units_in_tick), depending on whether the access unit
is coded as a frame picture or field picture, and― Using these inferences, the CPB will not overflow or underflow and the DPB will not overflow.
6.5.7 Decoder conformance test of a particular profile and levelIn order for a decoder of a particular profile and level to claim output order conformance to ISO/
IEC 23094-1 as specified by this document, the decoder should successfully pass the static test specified
in subclause 6.5.5 with all the bitstreams of the normative test suite specified for testing decoders of
this particular profile and level combination.In order for a decoder of a particular profile and level to claim output timing conformance to ISO/
IEC 23094-1 as specified by this document, the decoder should successfully pass both the static test
specified in subclause 6.5.5 and the dynamic test specified in subclause 6.5.6 with all the bitstreams of
the normative test suite specified for testing decoders of this particular profile and level. Table 1 specify
the normative test suites for each profile and level combination. The test suite for a particular profile
and level combination is the list of bitstreams that are marked with an 'X' in the column corresponding
to that profile and level combination.'X' indicates that the bitstream is designed to test both the dynamic and static conformance of the
decoder.The bitstream column specifies the bitstream used for each test.
A decoder that conforms to the Baseline profile or Main profile with 10 bit-depth at a specific level
should be capable of decoding the specified bitstreams in Table 1.6.6 Specification of the test bitstreams
6.6.1 General
Some characteristics of each bitstream listed in Table 1 are specified in this clause. In Table 1, the
value "29,97" should be interpreted as an approximation of an exact value of 30 000 ÷ 1 001 and the
value "59,94" should be interpreted as an approximation of an exact value of 60 000 ÷ 1 001.
A set of test vectors have been provided for conformance and are available at the following address:
― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/© ISO/IEC 2021 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
6.6.2 Test bitstreams
6.6.2.1 Test bitstream BP_SET_A
Specification: Streams with sets of coding tools in Baseline profile.
Functional stage: Test the decoding process of Baseline profile.
Purpose: Check that the decoder can properly decode bitstreams in which the full set of coding tools in
Baseline profile is enabled.6.6.2.2 Test bitstream BP_SET_B
Specification: Streams with sets of coding tools in Baseline Still Picture profile.
Functional stage: Test the decoding process of Baseline Still Picture profile.Purpose: Check that the decoder can properly decode bitstreams in which the full set of coding tools in
Baseline Still Picture profile is enabled.6.6.2.3 Test bitstream MP_MIN_A
Specification: Streams with all tools disabled in Main profile in which SPS tool flags are equal to 0 in
Main profile.Functional stage: Test the decoding process of Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools
are disabled in Main profile.6.6.2.4 Test bitstream MP_MIN_B
Specification: Streams with all tools disabled in Main Still Picture profile in which SPS tool flags are
equal to 0.Functional stage: Test the decoding process of Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools
are disabled in Main Still Picture profile6.6.2.5 Test bitstream MP_SET_A
Specification: Streams with all tools enabled in Main profile in which SPS tool flags are equal to 1 in
Main profile.Functional stage: Test the decoding process of Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools
are enabled in Main profile.6.6.2.6 Test bitstream MP_SET_B
Specification: Streams with all tools enabled in Main Still Picture profile in which SPS tool flags are
equal to 1 in Main profile.Functional stage: Test the decoding process of Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools
are enabled in Main Still Picture profile.© ISO/IEC 2021 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
6.6.2.7 Test bitstream CTU_A
Specification: Streams with max CTU size of 64 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 64 is enabled
in Main profile.6.6.2.8 Test bitstream CTU_B
Specification: Streams with max CTU size of 32 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 32 is enabled
in Main profile.6.6.2.9 Test bitstream CTU_C
Specification: Streams with max CTU size of 128 and min CTU size of 8 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 128 and min
CTU size of 8 are enabled in Main profile.6.6.2.10 Test bitstream CTU_D
Specification: Streams with CTU size of 32 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.
Purpose: Check that the decoder can properly decode bitstreams in which CTU size of 32 is enabled in
Main profile.6.6.2.11 Test bitstream CTU_E
Specification: Streams with max CTU size of 64 enabled in Main Still Picture profile.
Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 64 is enabled
in Main Still Picture profile.6.6.2.12 Test bitstream CTU_F
Specification: Streams with max CTU size of 32 enabled in Main Still Picture profile.
Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 32 is enabled
in Main Still Picture profile.6.6.2.13 Test bitstream CTU_G
Specification: Streams with max CTU size of 128 and min CTU size of 8 enabled in Main Still Picture
profile.Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.
© ISO/IEC 2021 – All rights reserved---------------------- Page: 11 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 128 and min
CTU size of 8 are enabled in Main Still Picture profile.6.6.2.14 Test bitstream CTU_H
Specification: Streams with CTU size of 32 enabled in Main Still Picture profile.
Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.
Purpose: Check that the decoder can properly decode bitstreams in which CTU size of 32 is enabled in
Main Still Picture profile.6.6.2.15 Test bitstream BTT_A
Specification: Streams with B
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.