Information technology — General video coding — Part 4: Conformance and reference software for essential video coding

This document specifies a set of tests and procedures designed to indicate whether encoders or decoders essential video coding (EVC), which contains tests and a reference software designed to verify whether bitstreams and decoders meet normative requirements specified in ISO/IEC 23094-1. An encoder can claim conformance to ISO/IEC 23094-1 if the bitstreams that it generates are conforming bitstreams. Characteristics of coded bitstreams and decoders are defined in ISO/IEC 23094-1. Decoder characteristics define the properties and capabilities of the applied decoding process. The capabilities of a decoder specify which bitstreams the decoder can decode and reconstruct. A bitstream can be decoded by a decoder if the characteristics of the bitstream are within the specified decoder capabilities.

Technologies de l'information — Codage vidéo général — Partie 4: Conformité et logiciel de référence pour le codage vidéo essentiel

General Information

Status
Published
Publication Date
07-Feb-2022
Current Stage
5060 - Close of voting Proof returned by Secretariat
Completion Date
01-Dec-2021
Ref Project

Buy Standard

Standard
ISO/IEC 23094-4:2022 - Information technology — General video coding — Part 4: Conformance and reference software for essential video coding Released:2/8/2022
English language
32 pages
sale 15% off
Preview
sale 15% off
Preview
Draft
ISO/IEC FDIS 23094-4 - Information technology -- General video coding
English language
32 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

INTERNATIONAL ISO/IEC
STANDARD 23094-4
First edition
2022-01
Information technology — General
video coding —
Part 4:
Conformance and reference software
for essential video coding
Technologies de l'information — Codage vidéo général —
Partie 4: Conformité et logiciel de référence pour le codage vidéo
essentiel
Reference number
ISO/IEC 23094-4:2022(E)
© ISO/IEC 2022
---------------------- Page: 1 ----------------------
ISO/IEC 23094-4:2022(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2022

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on

the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below

or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 23094-4:2022(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ..................................................................................................................................................................................... 1

3 Terms and definitions .................................................................................................................................................................................... 1

4 Abbreviated terms ............................................................................................................................................................................................. 2

5 Conventions ............................................................................................................................................................................................................... 2

6 Conformance testing for ISO/IEC 23094-1 ................................................................................................................................ 2

6.1 Introduction .............................................................................................................................................................................................. 2

6.2 Bitstream conformance .................................................................................................................................................................. 2

6.3 Decoder conformance ...................................................................................................................................................................... 2

6.4 Procedure to test bitstreams ..................................................................................................................................................... 2

6.5 Procedure to test decoder conformance ......................................................................................................................... 3

6.5.1 Conformance bitstreams ............................................................................................................................................ 3

6.5.2 Contents of the bitstream file ................................................................................................................................. 3

6.5.3 Requirements on output of the decoding process and timing ................................................... 3

6.5.4 Recommendations............................................................................................................................................................. 4

6.5.5 Static tests for output order conformance .................................................................................................. 4

6.5.6 Dynamic tests for output timing conformance ....................................................................................... 4

6.5.7 Decoder conformance test of a particular profile and level ........................................................ 5

6.6 Specification of the test bitstreams ..................................................................................................................................... 5

6.6.1 General ........................................................................................................................................................................................ 5

6.6.2 Test bitstreams .................................................................................................................................................................... 6

6.7 Test suites for ISO/IEC 23094-1 ...........................................................................................................................................25

7 Reference software description .......................................................................................................................................................30

7.1 General ........................................................................................................................................................................................................30

7.2 ETM repository ................................................................................................................................................................................... 31

7.3 Encoder and decoder usage ..................................................................................................................................................... 31

iii
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 3 ----------------------
ISO/IEC 23094-4:2022(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are

members of ISO or IEC participate in the development of International Standards through technical

committees established by the respective organization to deal with particular fields of technical

activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international

organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the

work.

The procedures used to develop this document and those intended for its further maintenance

are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria

needed for the different types of document should be noted. This document was drafted in

accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or

www.iec.ch/members_experts/refdocs).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC

list of patent declarations received (see https://patents.iec.ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to

the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see

www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.

A list of all parts in the ISO/IEC 23094 series can be found on the ISO website and IEC websites.

Any feedback or questions on this document should be directed to the user’s national standards

body. A complete listing of these bodies can be found at www.iso.org/members.html and

www.iec.ch/national-committees.
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 4 ----------------------
INTERNATIONAL STANDARD ISO/IEC 23094-4:2022(E)
Information technology — General video coding —
Part 4:
Conformance and reference software for essential video
coding
1 Scope

This document specifies a set of tests and procedures designed to indicate whether encoders or

decoders essential video coding (EVC), which contains tests and a reference software designed to verify

whether bitstreams and decoders meet normative requirements specified in ISO/IEC 23094-1. An

encoder can claim conformance to ISO/IEC 23094-1 if the bitstreams that it generates are conforming

bitstreams. Characteristics of coded bitstreams and decoders are defined in ISO/IEC 23094-1. Decoder

characteristics define the properties and capabilities of the applied decoding process. The capabilities

of a decoder specify which bitstreams the decoder can decode and reconstruct. A bitstream can be

decoded by a decoder if the characteristics of the bitstream are within the specified decoder capabilities.

2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO/IEC 23094-1:2020, Information technology — General video coding — Part 1: Essential video coding

ISO/IEC 9899, Information technology — Programming languages — C

ISO/IEC/IEEE 9945, Information technology — Portable Operating System Interface (POSIX®) Base

Specifications, Issue 7
3 Terms and definitions

For the purposes of this document, the terms and definitions and symbols specified in ISO/IEC 23094-1

and the following apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
bitstream

sequence of bits, in the form of a NAL unit stream or a raw bitstream, that forms the representation of

coded pictures and associated data forming one or more coded video sequences
3.2
decoder

embodiment of a process that operates on a bitstream and may conform to the decoding process

requirements specified for conformance

Note 1 to entry: The decoder does not include the display process, which is outside the scope of this document

© ISO/IEC 2022 – All rights reserved
---------------------- Page: 5 ----------------------
ISO/IEC 23094-4:2022(E)
3.3
encoder
embodiment of a process that produces a bitstream

Note 1 to entry: The process is not specified in this document (except in regard to identification of the reference

software encoder).
3.4
reference software decoder

software which may decode a bitstream (3.1) encoded according to the syntax structure

4 Abbreviated terms
ETM test model of essential video coding
HRD hypothetical reference decoder
5 Conventions

For the purposes of this document, relevant conventions are specified in Clause 5 of

ISO/IEC 23094-1:2020.
6 Conformance testing for ISO/IEC 23094-1
6.1 Introduction

The following subclauses specify the tests for verifying conformance of video bitstreams as well as

decoders. These tests make use of test data (bitstream test suites) provided as an electronic annex to

this document and the reference software decoder specified in ISO/IEC 23094-1.
The electronic annex to this document is available at the following address:
― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/
6.2 Bitstream conformance
Bitstream conformance is specified by Clause C.4 of ISO/IEC 23094-1:2020.
6.3 Decoder conformance
Decoder conformance is specified by Clause C.5 of ISO/IEC 23094-1:2020.
6.4 Procedure to test bitstreams

A bitstream that claims conformance with ISO/IEC 23094-1 should pass the following normative test.

The bitstream should be decoded by processing it with the reference software decoder. When processed

by the reference software decoder, the bitstream should not cause any error or non-conformance

messages to be reported by the reference software decoder. This test should not be applied to bitstreams

that are known to contain errors introduced by transmission, as such errors are highly likely to result

in bitstreams that lack conformance to ISO/IEC 23094-1.

Successfully passing the reference software decoder test provides only a strong presumption that the

bitstream under test is conforming to the video layer, i.e., that it does indeed meet all the requirements

for the video layer (except Annexes C, D and E) specified in ISO/IEC 23094-1:2020 that are tested by the

reference software decoder.
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC 23094-4:2022(E)

Additional tests may be necessary to more thoroughly check that the bitstream properly meets all the

requirements specified in ISO/IEC 23094-1:2020 including the hypothetical reference decoder (HRD)

conformance (based on Annexes C, D and E). These complementary tests may be performed using other

video bitstream verifiers that perform more complete tests than those implemented by the reference

software decoder.

When testing a bitstream for conformance, it may also be useful to test whether or not the bitstream

follows the informative recommendations specified in ISO/IEC 23094-1.

To check correctness of a bitstream, it is necessary to parse the entire bitstream and to extract all the

syntax elements and other values derived from those syntactic elements and used by the decoding

process specified in ISO/IEC 23094-1.

A verifier may not necessarily perform all stages of the decoding process specified in ISO/IEC 23094-1

in order to verify bitstream correctness. Many tests can be performed on syntax elements in a state

prior to their use in some processing stages.
6.5 Procedure to test decoder conformance
6.5.1 Conformance bitstreams

A bitstream has values of profile_idc and level_idc corresponding to a set of specified constraints on

a bitstream for which a decoder conforming to a specified profile and level is required in Annex A of

ISO/IEC 23094-1:2020 to properly perform the decoding process.
6.5.2 Contents of the bitstream file

The conformance bitstreams are included in this document as an electronic attachment available at the

following address:
― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/

The following information is included in a single zipped file for each such bitstream.

― bitstream, and
― decoded pictures or hashes of decoded pictures (may not be present), and
― short description of the bitstream, and
― trace file (results while decoding the bitstream, in ASCII format).

In cases where the decoded pictures or hashes of decoded pictures are not available, the reference

software decoder should be used to generate the necessary reference decoded pictures from the

bitstream.
6.5.3 Requirements on output of the decoding process and timing
Two classes of decoder conformance are specified:
― output order conformance, and
― output timing conformance.

The output of the decoding process is specified in Clause 8 and Annex C of ISO/IEC 23094-1:2020.

For output order conformance, it is a requirement that all of the decoded pictures specified for output

in Annex C of ISO/IEC 23094-1:2020 should be output by a conforming decoder in the specified order

and that the values of the decoded samples in all of the pictures that are output should be (exactly equal

to) the values specified in Clause 8 of ISO/IEC 23094-1:2020.
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC 23094-4:2022(E)

For output timing conformance, it is a requirement that a conforming decoder should also output the

decoded samples at the rates and times specified in Annex C of ISO/IEC 23094-1:2020.

The display process, which ordinarily follows the output of the decoding process, is outside the scope of

this International Standard.
6.5.4 Recommendations

In addition to the requirements, it is desirable that conforming decoders implement various

informative recommendations specified in ISO/IEC 23094-1. This subclause discusses some of these

recommendations.

It is recommended that a conforming decoder be able to resume the decoding process as soon as possible

after the loss or corruption of part of a bitstream. In most cases it is possible to resume decoding at the

next slice header. It is recommended that a conforming decoder be able to perform concealment for the

coding tree blocks or video packets for which all the coded data has not been received.

6.5.5 Static tests for output order conformance

Static tests of a video decoder require testing of the decoded samples. This clause will explain how

this test can be accomplished when the decoded samples at the output of the decoding process are

available. It may not be possible to perform this type of test with a production decoder (due to the

lack of an appropriate accessible interface in the design at which to perform the test). In that case this

test should be performed by the manufacturer during the design and development phase. Static tests

are used for testing the decoding process. The test will check that the values of the samples decoded

by the decoder under test should be identical to the values of the samples decoded by the reference

decoder. When a hash of the values of the samples of the decoded pictures is attached to the bitstream

file, a corresponding hash operation performed on the values of the samples of the decoded pictures

produced by the decoder under test should produce the same results.
6.5.6 Dynamic tests for output timing conformance

Dynamic tests are applied to check that all the decoded samples are output and that the timing of

the output of the decoder's decoded samples conforms to the specification of Clause 8 and Annex C of

ISO/IEC 23094-1:2020, and to verify that the HRD models (as specified by the CPB and DPB specification

in Annex C of ISO/IEC 23094-1:2020) are not violated when the bits of the bitstream are delivered at the

proper rate.

The dynamic test is often easier to perform on a complete decoding system, which may include a

systems decoder, a video decoder and a display process. It may be possible to record the output of the

display process and to check that display order and timing of decoded pictures are correct at the output

of the display process. However, since the display process is not within the scope of ISO/IEC 23094-1,

there may be cases where the output of the display process differs in timing or value even though the

video decoder is conforming. In this case, the output of the video decoder itself (before the display

process) would need to be captured in order to perform the dynamic tests on the video decoder. In

particular the output order and timing of the decoded pictures should be correct.

If buffering period and picture timing SEI messages are included in the test bitstream, HRD conformance

should be verified using the values of initial_cpb_removal_delay, initial_cpb_removal_delay_offset, cpb_

removal_delay and dpb_removal_delay that are included in the bitstream.

If buffering period and picture timing SEI messages are not included in the bitstream, the following

inferences should be made to generate the missing parameters:
― fixed_pic_rate_flag should be inferred to be equal to 1, and
― low_delay_hrd_flag should be inferred to be equal to 0, and
― cbr_flag should be inferred to be equal to 0, and
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 23094-4:2022(E)

― The frame rate of the bitstream should be inferred to be equal to the frame rate value specified in

the corresponding Table 1 of 6.7, where the bitstream is listed. If this is missing, then a frame rate of

either 25 or 30 000 ÷ 1 001 can be inferred, and

― time_scale should be set equal to 90 000 and the value of num_units_in_tick should be computed

based on frame rate, and

― The bit rate of the bitstream should be inferred to be equal to the maximum value for the level

specified in Table A.1 in ISO/IEC 23094-1:2020, and

― CPB and DPB sizes should be inferred to be equal to the maximum value for the level specified in

Table A.1 in ISO/IEC 23094-1:2020.
With the above inferences, the HRD should be operated as follows:

― The CPB is filled starting at time t = 0, until it is full, before removal of the first access unit. This

means that the initial_cpb_removal_delay should be inferred to be equal to the total CPB buffer size

divided by the bit rate divided by 90 000 (rounded downwards) and initial_cpb_removal_delay_

offset should be inferred to be equal to zero, and

― The first access unit is removed at time t = initial_cpb_removal_delay ÷ 90 000 and subsequent

access units are removed at intervals based on the frame distance, i.e., 2 * (90 000 ÷ num_units_in_

tick) or the field distance, i.e., (90 000 / num_units_in_tick), depending on whether the access unit

is coded as a frame picture or field picture, and

― Using these inferences, the CPB will not overflow or underflow and the DPB will not overflow.

6.5.7 Decoder conformance test of a particular profile and level

In order for a decoder of a particular profile and level to claim output order conformance to

ISO/IEC 23094-1 as specified by this document, the decoder should successfully pass the static test

specified in subclause 6.5.5 with all the bitstreams of the normative test suite specified for testing

decoders of this particular profile and level combination.

In order for a decoder of a particular profile and level to claim output timing conformance to

ISO/IEC 23094-1 as specified by this document, the decoder should successfully pass both the static test

specified in subclause 6.5.5 and the dynamic test specified in subclause 6.5.6 with all the bitstreams of

the normative test suite specified for testing decoders of this particular profile and level. Table 1 specify

the normative test suites for each profile and level combination. The test suite for a particular profile

and level combination is the list of bitstreams that are marked with an 'X' in the column corresponding

to that profile and level combination.

'X' indicates that the bitstream is designed to test both the dynamic and static conformance of the

decoder.
The bitstream column specifies the bitstream used for each test.

A decoder that conforms to the Baseline profile or Main profile with 10 bit-depth at a specific level

should be capable of decoding the specified bitstreams in Table 1.
6.6 Specification of the test bitstreams
6.6.1 General

Some characteristics of each bitstream listed in Table 1 are specified in this clause. In Table 1, the

value "29,97" should be interpreted as an approximation of an exact value of 30 000 ÷ 1 001 and the

value "59,94" should be interpreted as an approximation of an exact value of 60 000 ÷ 1 001.

A set of test vectors have been provided for conformance and are available at the following address:

― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC 23094-4:2022(E)
6.6.2 Test bitstreams
6.6.2.1 Test bitstream BP_SET_A
Specification: Streams with sets of coding tools in Baseline profile.
Functional stage: Test the decoding process of Baseline profile.

Purpose: Check that the decoder can properly decode bitstreams in which the full set of coding tools in

Baseline profile is enabled.
6.6.2.2 Test bitstream BP_SET_B

Specification: Streams with sets of coding tools in Baseline Still Picture profile.

Functional stage: Test the decoding process of Baseline Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which the full set of coding tools in

Baseline Still Picture profile is enabled.
6.6.2.3 Test bitstream MP_MIN_A

Specification: Streams with all tools disabled in Main profile in which SPS tool flags are equal to 0 in

Main profile.
Functional stage: Test the decoding process of Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools

are disabled in Main profile.
6.6.2.4 Test bitstream MP_MIN_B

Specification: Streams with all tools disabled in Main Still Picture profile in which SPS tool flags are

equal to 0.
Functional stage: Test the decoding process of Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools

are disabled in Main Still Picture profile
6.6.2.5 Test bitstream MP_SET_A

Specification: Streams with all tools enabled in Main profile in which SPS tool flags are equal to 1 in

Main profile.
Functional stage: Test the decoding process of Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools

are enabled in Main profile.
6.6.2.6 Test bitstream MP_SET_B

Specification: Streams with all tools enabled in Main Still Picture profile in which SPS tool flags are

equal to 1 in Main profile.
Functional stage: Test the decoding process of Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools

are enabled in Main Still Picture profile.
© ISO/IEC 2022 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC 23094-4:2022(E)
6.6.2.7 Test bitstream CTU_A
Specification: Streams with max CTU size of 64 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 64 is enabled

in Main profile.
6.6.2.8 Test bitstream CTU_B
Specification: Streams with max CTU size of 32 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 32 is enabled

in Main profile.
6.6.2.9 Test bitstream CTU_C

Specification: Streams with max CTU size of 128 and min CTU size of 8 enabled in Main profile.

Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 128 and min

CTU size of 8 are enabled in Main profile.
6.6.2.10 Test bitstream CTU_D
Specification: Streams with CTU size of 32 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which CTU size of 32 is enabled in

Main profile.
6.6.2.11 Test bitstream CTU_E

Specification: Streams with max CTU size of 64 enabled in Main Still Picture profile.

Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 64 is enabled

in Main Still Picture profile.
6.6.2.12 Test bitstream CTU_F

Specification: Streams with max CTU size of 32 enabled in Main Still Picture profile.

Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 32 is enabled

in Main Still Picture profile.
6.6.2.13 Test bitstream CTU_G

Specification: Streams with max CTU size of 128 and min CTU size of 8 enabled in Main Still Picture

profile.

Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.

© ISO/IEC 2022 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC 23094-4:2022(E)

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 128 and min

CTU size of 8 are enabled in Main Still Picture profile.
6.6.2.14 Test bitstream CTU_H

Specification: Streams with CTU size of 32 enabled in Main Still Picture profile.

Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which CTU size of 32 is enabled in

Main Still Picture profile.
6.6.2.15 Test bitstream BTT_A
Specification: Streams with BTT tool disabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which BTT tool is disabled in Main

profile.
6.6.2.16 Test bitstream BTT_B
Specification: Streams with only BTT tool enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which only BTT tool is enabled in

Main profile.
...

FINAL
INTERNATIONAL ISO/IEC
DRAFT
STANDARD FDIS
23094-4
ISO/IEC JTC 1/SC 29
Information technology — General
Secretariat: JISC
video coding —
Voting begins on:
2021-10-05
Part 4:
Voting terminates on:
Conformance and reference software
2021-11-30
for essential video coding
RECIPIENTS OF THIS DRAFT ARE INVITED TO
SUBMIT, WITH THEIR COMMENTS, NOTIFICATION
OF ANY RELEVANT PATENT RIGHTS OF WHICH
THEY ARE AWARE AND TO PROVIDE SUPPOR TING
DOCUMENTATION.
IN ADDITION TO THEIR EVALUATION AS
Reference number
BEING ACCEPTABLE FOR INDUSTRIAL, TECHNO-
ISO/IEC FDIS 23094-4:2021(E)
LOGICAL, COMMERCIAL AND USER PURPOSES,
DRAFT INTERNATIONAL STANDARDS MAY ON
OCCASION HAVE TO BE CONSIDERED IN THE
LIGHT OF THEIR POTENTIAL TO BECOME STAN-
DARDS TO WHICH REFERENCE MAY BE MADE IN
NATIONAL REGULATIONS. © ISO/IEC 2021
---------------------- Page: 1 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2021

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on

the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below

or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ..................................................................................................................................................................................... 1

3 Terms and definitions .................................................................................................................................................................................... 1

4 Abbreviated terms ............................................................................................................................................................................................. 2

5 Conventions ............................................................................................................................................................................................................... 2

6 Conformance testing for ISO/IEC 23094-1 ................................................................................................................................ 2

6.1 Introduction .............................................................................................................................................................................................. 2

6.2 Bitstream conformance .................................................................................................................................................................. 2

6.3 Decoder conformance ...................................................................................................................................................................... 2

6.4 Procedure to test bitstreams ..................................................................................................................................................... 2

6.5 Procedure to test decoder conformance ......................................................................................................................... 3

6.5.1 Conformance bitstreams ............................................................................................................................................ 3

6.5.2 Contents of the bitstream file ................................................................................................................................. 3

6.5.3 Requirements on output of the decoding process and timing ................................................... 3

6.5.4 Recommendations............................................................................................................................................................. 4

6.5.5 Static tests for output order conformance .................................................................................................. 4

6.5.6 Dynamic tests for output timing conformance ....................................................................................... 4

6.5.7 Decoder conformance test of a particular profile and level ........................................................ 5

6.6 Specification of the test bitstreams ..................................................................................................................................... 5

6.6.1 General ........................................................................................................................................................................................ 5

6.6.2 Test bitstreams .................................................................................................................................................................... 6

6.7 Test suites for ISO/IEC 23094-1 ...........................................................................................................................................25

7 Reference software description .......................................................................................................................................................30

7.1 General ........................................................................................................................................................................................................30

7.2 ETM repository ................................................................................................................................................................................... 31

7.3 Encoder and decoder usage ..................................................................................................................................................... 31

iii
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 3 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that are

members of ISO or IEC participate in the development of International Standards through technical

committees established by the respective organization to deal with particular fields of technical

activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international

organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the

work.

The procedures used to develop this document and those intended for its further maintenance

are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria

needed for the different types of document should be noted. This document was drafted in

accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or

www.iec.ch/members_experts/refdocs).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC

list of patent declarations received (see https://patents.iec.ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to

the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see

www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.

A list of all parts in the ISO/IEC 23094 series can be found on the ISO website and IEC websites.

Any feedback or questions on this document should be directed to the user’s national standards

body. A complete listing of these bodies can be found at www.iso.org/members.html and

www.iec.ch/national-committees.
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 4 ----------------------
FINAL DRAFT INTERNATIONAL STANDARD ISO/IEC FDIS 23094-4:2021(E)
Information technology — General video coding —
Part 4:
Conformance and reference software for essential video
coding
1 Scope

This document specifies a set of tests and procedures designed to indicate whether encoders or

decoders essential video coding (EVC), which contains tests and a reference software designed to verify

whether bitstreams and decoders meet normative requirements specified in ISO/IEC 23094-1. An

encoder can claim conformance to ISO/IEC 23094-1 if the bitstreams that it generates are conforming

bitstreams. Characteristics of coded bitstreams and decoders are defined in ISO/IEC 23094-1. Decoder

characteristics define the properties and capabilities of the applied decoding process. The capabilities

of a decoder specify which bitstreams the decoder can decode and reconstruct. A bitstream can be

decoded by a decoder if the characteristics of the bitstream are within the specified decoder capabilities.

2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO/IEC 23094-1:2020, Information technology — General video coding — Part 1: Essential video coding

ISO/IEC 9899, Information technology — Programming languages — C

ISO/IEC/IEEE 9945, Information technology — Portable Operating System Interface (POSIX®) Base

Specifications, Issue 7
3 Terms and definitions

For the purposes of this document, the terms and definitions and symbols specified in ISO/IEC 23094-1

and the following apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
bitstream

sequence of bits, in the form of a NAL unit stream or a raw bitstream, that forms the representation of

coded pictures and associated data forming one or more coded video sequences
3.2
decoder

embodiment of a process that operates on a bitstream and may conform to the decoding process

requirements specified for conformance

Note 1 to entry: The decoder does not include the display process, which is outside the scope of this document

© ISO/IEC 2021 – All rights reserved
---------------------- Page: 5 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
3.3
encoder
embodiment of a process that produces a bitstream

Note 1 to entry: The process is not specified in this document (except in regard to identification of the reference

software encoder).
3.4
reference software decoder

software which may decode a bitstream (3.1) encoded according to the syntax structure

4 Abbreviated terms
ETM test model of essential video coding
HRD hypothetical reference decoder
5 Conventions

For the purposes of this document, relevant conventions are specified in Clause 5 of ISO/IEC 23094-

1:2020.
6 Conformance testing for ISO/IEC 23094-1
6.1 Introduction

The following subclauses specify the tests for verifying conformance of video bitstreams as well as

decoders. These tests make use of test data (bitstream test suites) provided as an electronic annex to

this document and the reference software decoder specified in ISO/IEC 23094-1.
The electronic annex to this document is available at the following address:
― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/
6.2 Bitstream conformance
Bitstream conformance is specified by Clause C.4 of ISO/IEC 23094-1:2020.
6.3 Decoder conformance
Decoder conformance is specified by Clause C.5 of ISO/IEC 23094-1:2020.
6.4 Procedure to test bitstreams

A bitstream that claims conformance with ISO/IEC 23094-1 should pass the following normative test.

The bitstream should be decoded by processing it with the reference software decoder. When processed

by the reference software decoder, the bitstream should not cause any error or non-conformance

messages to be reported by the reference software decoder. This test should not be applied to bitstreams

that are known to contain errors introduced by transmission, as such errors are highly likely to result

in bitstreams that lack conformance to ISO/IEC 23094-1.

Successfully passing the reference software decoder test provides only a strong presumption that the

bitstream under test is conforming to the video layer, i.e., that it does indeed meet all the requirements

for the video layer (except Annexes C, D and E) specified in ISO/IEC 23094-1:2020 that are tested by the

reference software decoder.
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC FDIS 23094-4:2021(E)

Additional tests may be necessary to more thoroughly check that the bitstream properly meets all the

requirements specified in ISO/IEC 23094-1:2020 including the hypothetical reference decoder (HRD)

conformance (based on Annexes C, D and E). These complementary tests may be performed using other

video bitstream verifiers that perform more complete tests than those implemented by the reference

software decoder.

When testing a bitstream for conformance, it may also be useful to test whether or not the bitstream

follows the informative recommendations specified in ISO/IEC 23094-1.

To check correctness of a bitstream, it is necessary to parse the entire bitstream and to extract all the

syntax elements and other values derived from those syntactic elements and used by the decoding

process specified in ISO/IEC 23094-1.

A verifier may not necessarily perform all stages of the decoding process specified in ISO/IEC 23094-1

in order to verify bitstream correctness. Many tests can be performed on syntax elements in a state

prior to their use in some processing stages.
6.5 Procedure to test decoder conformance
6.5.1 Conformance bitstreams

A bitstream has values of profile_idc and level_idc corresponding to a set of specified constraints on a

bitstream for which a decoder conforming to a specified profile and level is required in Annex A of ISO/

IEC 23094-1:2020 to properly perform the decoding process.
6.5.2 Contents of the bitstream file

The conformance bitstreams are included in this document as an electronic attachment available at the

following address:
― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/

The following information is included in a single zipped file for each such bitstream.

― bitstream, and
― decoded pictures or hashes of decoded pictures (may not be present), and
― short description of the bitstream, and
― trace file (results while decoding the bitstream, in ASCII format).

In cases where the decoded pictures or hashes of decoded pictures are not available, the reference

software decoder should be used to generate the necessary reference decoded pictures from the

bitstream.
6.5.3 Requirements on output of the decoding process and timing
Two classes of decoder conformance are specified:
― output order conformance, and
― output timing conformance.

The output of the decoding process is specified in Clause 8 and Annex C of ISO/IEC 23094-1:2020.

For output order conformance, it is a requirement that all of the decoded pictures specified for output

in Annex C of ISO/IEC 23094-1:2020 should be output by a conforming decoder in the specified order

and that the values of the decoded samples in all of the pictures that are output should be (exactly equal

to) the values specified in Clause 8 of ISO/IEC 23094-1:2020.
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 7 ----------------------
ISO/IEC FDIS 23094-4:2021(E)

For output timing conformance, it is a requirement that a conforming decoder should also output the

decoded samples at the rates and times specified in Annex C of ISO/IEC 23094-1:2020.

The display process, which ordinarily follows the output of the decoding process, is outside the scope of

this International Standard.
6.5.4 Recommendations

In addition to the requirements, it is desirable that conforming decoders implement various

informative recommendations specified in ISO/IEC 23094-1. This subclause discusses some of these

recommendations.

It is recommended that a conforming decoder be able to resume the decoding process as soon as possible

after the loss or corruption of part of a bitstream. In most cases it is possible to resume decoding at the

next slice header. It is recommended that a conforming decoder be able to perform concealment for the

coding tree blocks or video packets for which all the coded data has not been received.

6.5.5 Static tests for output order conformance

Static tests of a video decoder require testing of the decoded samples. This clause will explain how

this test can be accomplished when the decoded samples at the output of the decoding process are

available. It may not be possible to perform this type of test with a production decoder (due to the

lack of an appropriate accessible interface in the design at which to perform the test). In that case this

test should be performed by the manufacturer during the design and development phase. Static tests

are used for testing the decoding process. The test will check that the values of the samples decoded

by the decoder under test should be identical to the values of the samples decoded by the reference

decoder. When a hash of the values of the samples of the decoded pictures is attached to the bitstream

file, a corresponding hash operation performed on the values of the samples of the decoded pictures

produced by the decoder under test should produce the same results.
6.5.6 Dynamic tests for output timing conformance

Dynamic tests are applied to check that all the decoded samples are output and that the timing of the

output of the decoder's decoded samples conforms to the specification of Clause 8 and Annex C of ISO/

IEC 23094-1:2020, and to verify that the HRD models (as specified by the CPB and DPB specification in

Annex C of ISO/IEC 23094-1:2020) are not violated when the bits of the bitstream are delivered at the

proper rate.

The dynamic test is often easier to perform on a complete decoding system, which may include a

systems decoder, a video decoder and a display process. It may be possible to record the output of the

display process and to check that display order and timing of decoded pictures are correct at the output

of the display process. However, since the display process is not within the scope of ISO/IEC 23094-1,

there may be cases where the output of the display process differs in timing or value even though the

video decoder is conforming. In this case, the output of the video decoder itself (before the display

process) would need to be captured in order to perform the dynamic tests on the video decoder. In

particular the output order and timing of the decoded pictures should be correct.

If buffering period and picture timing SEI messages are included in the test bitstream, HRD conformance

should be verified using the values of initial_cpb_removal_delay, initial_cpb_removal_delay_offset, cpb_

removal_delay and dpb_removal_delay that are included in the bitstream.

If buffering period and picture timing SEI messages are not included in the bitstream, the following

inferences should be made to generate the missing parameters:
― fixed_pic_rate_flag should be inferred to be equal to 1, and
― low_delay_hrd_flag should be inferred to be equal to 0, and
― cbr_flag should be inferred to be equal to 0, and
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC FDIS 23094-4:2021(E)

― The frame rate of the bitstream should be inferred to be equal to the frame rate value specified in

the corresponding Table 1 of 6.7, where the bitstream is listed. If this is missing, then a frame rate of

either 25 or 30 000 ÷ 1 001 can be inferred, and

― time_scale should be set equal to 90 000 and the value of num_units_in_tick should be computed

based on frame rate, and

― The bit rate of the bitstream should be inferred to be equal to the maximum value for the level

specified in Table A.1 in ISO/IEC 23094-1:2020, and

― CPB and DPB sizes should be inferred to be equal to the maximum value for the level specified in

Table A.1 in ISO/IEC 23094-1:2020.
With the above inferences, the HRD should be operated as follows:

― The CPB is filled starting at time t = 0, until it is full, before removal of the first access unit. This

means that the initial_cpb_removal_delay should be inferred to be equal to the total CPB buffer size

divided by the bit rate divided by 90 000 (rounded downwards) and initial_cpb_removal_delay_

offset should be inferred to be equal to zero, and

― The first access unit is removed at time t = initial_cpb_removal_delay ÷ 90 000 and subsequent

access units are removed at intervals based on the frame distance, i.e., 2 * (90 000 ÷ num_units_in_

tick) or the field distance, i.e., (90 000 / num_units_in_tick), depending on whether the access unit

is coded as a frame picture or field picture, and

― Using these inferences, the CPB will not overflow or underflow and the DPB will not overflow.

6.5.7 Decoder conformance test of a particular profile and level

In order for a decoder of a particular profile and level to claim output order conformance to ISO/

IEC 23094-1 as specified by this document, the decoder should successfully pass the static test specified

in subclause 6.5.5 with all the bitstreams of the normative test suite specified for testing decoders of

this particular profile and level combination.

In order for a decoder of a particular profile and level to claim output timing conformance to ISO/

IEC 23094-1 as specified by this document, the decoder should successfully pass both the static test

specified in subclause 6.5.5 and the dynamic test specified in subclause 6.5.6 with all the bitstreams of

the normative test suite specified for testing decoders of this particular profile and level. Table 1 specify

the normative test suites for each profile and level combination. The test suite for a particular profile

and level combination is the list of bitstreams that are marked with an 'X' in the column corresponding

to that profile and level combination.

'X' indicates that the bitstream is designed to test both the dynamic and static conformance of the

decoder.
The bitstream column specifies the bitstream used for each test.

A decoder that conforms to the Baseline profile or Main profile with 10 bit-depth at a specific level

should be capable of decoding the specified bitstreams in Table 1.
6.6 Specification of the test bitstreams
6.6.1 General

Some characteristics of each bitstream listed in Table 1 are specified in this clause. In Table 1, the

value "29,97" should be interpreted as an approximation of an exact value of 30 000 ÷ 1 001 and the

value "59,94" should be interpreted as an approximation of an exact value of 60 000 ÷ 1 001.

A set of test vectors have been provided for conformance and are available at the following address:

― https:// standards .iso .org/ iso -iec/ 23094/ -4/ ed -1/ en/
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 9 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
6.6.2 Test bitstreams
6.6.2.1 Test bitstream BP_SET_A
Specification: Streams with sets of coding tools in Baseline profile.
Functional stage: Test the decoding process of Baseline profile.

Purpose: Check that the decoder can properly decode bitstreams in which the full set of coding tools in

Baseline profile is enabled.
6.6.2.2 Test bitstream BP_SET_B

Specification: Streams with sets of coding tools in Baseline Still Picture profile.

Functional stage: Test the decoding process of Baseline Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which the full set of coding tools in

Baseline Still Picture profile is enabled.
6.6.2.3 Test bitstream MP_MIN_A

Specification: Streams with all tools disabled in Main profile in which SPS tool flags are equal to 0 in

Main profile.
Functional stage: Test the decoding process of Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools

are disabled in Main profile.
6.6.2.4 Test bitstream MP_MIN_B

Specification: Streams with all tools disabled in Main Still Picture profile in which SPS tool flags are

equal to 0.
Functional stage: Test the decoding process of Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools

are disabled in Main Still Picture profile
6.6.2.5 Test bitstream MP_SET_A

Specification: Streams with all tools enabled in Main profile in which SPS tool flags are equal to 1 in

Main profile.
Functional stage: Test the decoding process of Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools

are enabled in Main profile.
6.6.2.6 Test bitstream MP_SET_B

Specification: Streams with all tools enabled in Main Still Picture profile in which SPS tool flags are

equal to 1 in Main profile.
Functional stage: Test the decoding process of Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which all the SPS indicated tools

are enabled in Main Still Picture profile.
© ISO/IEC 2021 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC FDIS 23094-4:2021(E)
6.6.2.7 Test bitstream CTU_A
Specification: Streams with max CTU size of 64 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 64 is enabled

in Main profile.
6.6.2.8 Test bitstream CTU_B
Specification: Streams with max CTU size of 32 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 32 is enabled

in Main profile.
6.6.2.9 Test bitstream CTU_C

Specification: Streams with max CTU size of 128 and min CTU size of 8 enabled in Main profile.

Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 128 and min

CTU size of 8 are enabled in Main profile.
6.6.2.10 Test bitstream CTU_D
Specification: Streams with CTU size of 32 enabled in Main profile.
Functional stage: Test the decoding process of the partitioning in Main profile.

Purpose: Check that the decoder can properly decode bitstreams in which CTU size of 32 is enabled in

Main profile.
6.6.2.11 Test bitstream CTU_E

Specification: Streams with max CTU size of 64 enabled in Main Still Picture profile.

Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 64 is enabled

in Main Still Picture profile.
6.6.2.12 Test bitstream CTU_F

Specification: Streams with max CTU size of 32 enabled in Main Still Picture profile.

Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 32 is enabled

in Main Still Picture profile.
6.6.2.13 Test bitstream CTU_G

Specification: Streams with max CTU size of 128 and min CTU size of 8 enabled in Main Still Picture

profile.

Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.

© ISO/IEC 2021 – All rights reserved
---------------------- Page: 11 ----------------------
ISO/IEC FDIS 23094-4:2021(E)

Purpose: Check that the decoder can properly decode bitstreams in which max CTU size of 128 and min

CTU size of 8 are enabled in Main Still Picture profile.
6.6.2.14 Test bitstream CTU_H

Specification: Streams with CTU size of 32 enabled in Main Still Picture profile.

Functional stage: Test the decoding process of the partitioning in Main Still Picture profile.

Purpose: Check that the decoder can properly decode bitstreams in which CTU size of 32 is enabled in

Main Still Picture profile.
6.6.2.15 Test bitstream BTT_A
Specification: Streams with B
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.