ISO/IEC 23090-15:2022

Information technology — Coded representation of immersive media — Part 15: Conformance testing for versatile video coding

This document specifies a set of tests and procedures designed to indicate whether encoders or decoders meet the requirements specified in Rec. ITU‑T H.266 | ISO/IEC 23090-3.

Technologies de l'information — Représentation codée de média immersifs — Partie 15: Essai de conformité pour le codage vidéo polyvalent

General Information

• Status: Published
• Publication Date: 24-Nov-2022
• ICS: 35.040.40 - Coding of audio, video, multimedia and hypermedia information
• Technical Committee: ISO/IEC JTC 1/SC 29 - Coding of audio, picture, multimedia and hypermedia information
• Drafting Committee: ISO/IEC JTC 1/SC 29/WG 5 - MPEG Joint Video Coding Team(s) with ITU-T SG 16
• Current Stage: 9092 - International Standard to be revised
• Completion Date: 09-Feb-2023

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 23090-15
First edition
2022-11
Information technology — Coded
representation of immersive media —
Part 15:
Conformance testing for versatile
video coding
Technologies de l'information — Représentation codée de média
immersifs —
Partie 15: Essai de conformité pour le codage vidéo polyvalent
Reference number
ISO/IEC 23090-15:2022(E)
© ISO/IEC 2022

---------------------- Page: 1 ----------------------
ISO/IEC 23090-15:2022(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2022
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: [phone]
Email: [e-mail]
Website: www.iso.org
Published in Switzerland
ii
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC 23090-15:2022(E)
Contents Page
Foreword .iv
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms . 2
5 Conventions . 3
6 Conformance testing for ITU-T H.266 | ISO/IEC 23090-3 . 3
6.1 General . 3
6.2 Bitstream conformance . 3
6.3 Decoder conformance . 4
6.4 Procedure to test bitstreams . 4
6.5 Procedure to test decoder conformance . 4
6.5.1 Conformance bitstreams . 4
6.5.2 Contents of the bitstream file . 4
6.5.3 Requirements on output of the decoding process and timing . 5
6.5.4 Static tests for output order conformance . 5
6.5.5 Dynamic tests for output timing conformance . 6
6.5.6 Decoder conformance test for a particular profile, tier, and level . 6
6.6 Specification of the test bitstreams . 7
6.6.1 General . 7
6.6.2 Test bitstreams – Coding tools for Main 10 profile with 4:2:0 chroma
format and 10 bit depth . 7
6.6.3 Test bitstreams – High-level syntax features for Main 10 profile with 4:2:0
chroma format and 10 bit depth . 37
6.6.4 Test bitstreams – Additional chroma formats and bit depths for Main 10
profile . 50
6.6.5 Test bitstreams – Coding tools for Main 10 4:4:4 profile for 4:4:4 chroma
format and 10 bit depth . 51
6.6.6 Test bitstreams – Additional chroma formats and bit depths for Main 10
4:4:4 profile .56
6.6.7 Test bitstreams – Multilayer Main 10 profile.60
6.6.8 Test bitstreams – Multilayer Main 10 4:4:4 profile . 62
6.6.9 Test bitstreams – Main 10 Still Picture profile . 62
6.6.10 Test bitstreams – Main 10 4:4:4 Still Picture profile .63
6.7 Conformance test suites for Rec. ITU-T H.266 | ISO/IEC 23090-3 .63
6.7.1 Bitstreams for Main 10 profile.63
6.7.2 Bitstreams for Main 10 4:4:4 profile .68
6.7.3 Bitstreams for Multilayer Main 10 profile .68
6.7.4 Bitstreams for Multilayer Main 10 4:4:4 profile.68
6.7.5 Bitstreams for Main 10 Still Picture profile .68
6.7.6 Bitstreams for Main 10 4:4:4 Still Picture profile .69
iii
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 3 ----------------------
ISO/IEC 23090-15:2022(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance
are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria
needed for the different types of document should be noted. This document was drafted in
accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or
www.iec.ch/members_experts/refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC
list of patent declarations received (see https://patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see
www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information, in collaboration
with ITU-T Study Group 16 (as Rec. ITU-T H.266.1).
A list of all parts in the ISO/IEC 23090 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.
iv
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 4 ----------------------
INTERNATIONAL STANDARD ISO/IEC 23090-15:2022(E)
Information technology — Coded representation of
immersive media —
Part 15:
Conformance testing for versatile video coding
1 Scope
This document specifies a set of tests and procedures designed to indicate whether encoders or
decoders meet the requirements specified in Rec. ITU-T H.266 | ISO/IEC 23090-3.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
Rec. ITU-T H.266:2020 | ISO/IEC 23090-3:2021, Information technology – Coded representation of
immersive media– Part 3: Versatile video coding
Rec. ITU-T H.266.2 | ISO/IEC 23090-16, Information technology – Coded representation of immersive
media – Part 16: Reference software for versatile video coding
3 Terms and definitions
For the purposes of this document, the terms and definitions given in Rec. ITU-T H.266 | ISO/IEC 23090-3
and the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
bitstream
sequence of bits that conforms to specified syntax requirements or sequence of bits to be tested for
conformance to such syntax requirements
3.2
decoder
embodiment of a specified decoding process or process to be tested for conformance to such a decoding
process specification
3.3
encoder
embodiment of a process that produces a bitstream (3.1)
1
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 5 ----------------------
ISO/IEC 23090-15:2022(E)
3.4
reference software decoder
particular decoder (3.2) provided as a software package for use as an example available for study, as a
potential starting basis for the development of other decoders, as a way of testing bitstreams (3.1) for
conformance to a decoding process specification, or as a reference for comparison with the behaviour
of other decoders
3.5
reference software encoder
particular encoder (3.3) provided as a software package for use as an example available for study, as a
potential starting basis for the development of other encoders, or as a reference for comparison with
the behaviour of other encoders
4 Abbreviated terms
AMVP Adaptive motion vector prediction
CCLM Cross-component linear model
CIIP Combined inter/intra prediction
CST Chroma separate tree
CTC Common test conditions
DCT Discrete cosine transform
DMVR Decoder-side motion vector refinement
DQ Dependent quantization
DST Discrete sine transform
FTP File transfer protocol
ISP Intra subblock partitioning
JCCR Joint coding of chroma residuals
MMVD Merge with MVD
MPM Most probable mode
MRL Multiple reference line
AMVP Adaptive motion vector prediction
CCLM Cross-component linear model
CIIP Combined inter/intra prediction
CST Chroma separate tree
CTC Common test conditions
DCT Discrete cosine transform
DMVR Decoder-side motion vector refinement
DQ Dependent quantization
2
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 6 ----------------------
ISO/IEC 23090-15:2022(E)
DST Discrete sine transform
FTP File transfer protocol
ISP Intra subblock partitioning
JCCR Joint coding of chroma residuals
MMVD Merge with MVD
MPM Most probable mode
MRL Multiple reference line
MVD Motion vector difference
NUT NAL unit type
PDPC Position-dependent (intra) prediction combination
PERP Padded equirectangular projection
RPR Reference picture resampling
SAD Sum of absolute differences
SBT Subblock transform
SCC Screen content coding
SbTMVP Subblock based temporal motion vector prediction.
SMVD Symmetric MVD
TMVP Temporal motion vector prediction.
5 Conventions
The conventions in Clause 5 of Rec. ITU-T H.266:2020 | ISO/IEC 23090-3:2021 apply.
6 Conformance testing for ITU-T H.266 | ISO/IEC 23090-3
6.1 General
The conformance testing data for Rec. ITU-T H.266 | ISO/IEC 23090-3 is found in the electronic
attachment that can be obtained at the following location:
https:// standards .iso .org/ iso -iec/ 23090/ -15/ ed -1/ en/
The following subclauses specify normative tests for verifying conformance of video bitstreams as
well as decoders. Those normative tests make use of test data (bitstream test suites) provided as an
electronic attachment to this document and the reference software decoder specified in Rec. ITU-T
H.266.2 | ISO/IEC 23090-16.
6.2 Bitstream conformance
Bitstream conformance for Rec. ITU-T H.266 | ISO/IEC 23090-3 is specified by clause C.4 of Rec. ITU-T
H.266:2020 | ISO/IEC 23090-3:2021.
3
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC 23090-15:2022(E)
6.3 Decoder conformance
Decoder conformance for Rec. ITU-T H.266 | ISO/IEC 23090-3 is specified by clause C.5 of Rec. ITU-T
H.266:2020 | ISO/IEC 23090-3:2021.
6.4 Procedure to test bitstreams
A bitstream that is claimed to conform to Rec. ITU-T H.266 | ISO/IEC 23090-3 shall pass the following
normative test. This test should not be applied to bitstreams that are known to contain errors introduced
by transmission, as such errors are highly likely to result in bitstreams that lack conformance to Rec.
ITU-T H.266 | ISO/IEC 23090-3.
The bitstream under test shall be decoded by processing it with the reference software decoder
specified in Rec. ITU-T H.266.2 | ISO/IEC 23090-16. When processed by the reference software decoder,
the bitstream shall not cause any error or non-conformance messages to be reported by the reference
software decoder. When the bitstream under test contains decoded picture hash SEI messages, the hash
values signalled in the decoded picture hash SEI messages in the bitstream shall match those calculated
by the reference software decoder.
Successfully passing this test provides only a strong presumption that the bitstream under test does
indeed meet all the requirements specified in Rec. ITU-T H.266 | ISO/IEC 23090-3 that are tested by the
reference software decoder.
Additional tests may be necessary to more thoroughly check that the bitstream properly meets all
the requirements specified in Rec. ITU-T H.266 | ISO/IEC 23090-3, including hypothetical reference
decoder (HRD) conformance (based on Annexes C and D). Such complementary tests may be performed
using other video bitstream verifiers that perform more complete tests than those implemented by the
reference software decoder.
Rec. ITU-T H.266 | ISO/IEC 23090-3 contains several informative recommendations that are not
an integral part of that Recommendation | International Standard. When testing a bitstream for
conformance, it may also be useful to test whether or not the bitstream follows those recommendations.
To check the correctness of a bitstream, it is necessary to parse the entire bitstream and to extract all
the syntax elements and other values derived from those syntactic elements and used by the decoding
process specified in Rec. ITU-T H.266 | ISO/IEC 23090-3.
A bitstream verifier may not necessarily perform all stages of the decoding process specified in Rec.
ITU-T H.266 | ISO/IEC 23090-3 in order to verify bitstream correctness. Many tests can be performed
on syntax elements in a state prior to their use in some processing stages.
6.5 Procedure to test decoder conformance
6.5.1 Conformance bitstreams
A bitstream that conforms to Rec. ITU-T H.266 | ISO/IEC 23090-3 has values of general_profile_idc,
general_tier_flag, and general_level_idc corresponding to a set of specified constraints on a bitstream
for which a decoder conforming to a corresponding specified profile, tier, and level is required in
Annex A of Rec. ITU-T H.266:2020 | ISO/IEC 23090-3:2021 to properly perform the decoding process.
6.5.2 Contents of the bitstream file
The associated conformance testing bitstreams are included with this document as an electronic
attachment. The following information is included in a single zipped file for each such bitstream.
— *.bit – bitstream (provided for all bitstreams)
— *.txt – description (provided for all bitstreams)
— *.yuv.md5 – MD5 checksum of the complete decoded yuv file (provided for all bitstreams)
4
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC 23090-15:2022(E)
— *.md5 – MD5 checksum of the bitstream file (provided for all bitstreams)
— *.opl – output picture log (provided for all bitstreams)
— *.cfg – config file used to generate bitstream with VTM encoder software (not provided for all
bitstreams, not applicable if a VTM encoder release version was not used)
6.5.3 Requirements on output of the decoding process and timing
Two classes of decoder conformance are specified:
— output order conformance; and
— output timing conformance.
The output of the decoding process is specified in clause 8 and Annex C of Rec. ITU-T H.266:2020 |
ISO/IEC 23090-3:2021.
For output order conformance, it is a requirement that all of the cropped decoded pictures specified for
output in Annex C of Rec. ITU-T H.266:2020 | ISO/IEC 23090-3:2021 shall be output by a conforming
decoder in the specified order and that the values of the decoded samples of the cropped decoded
pictures that are output shall be (exactly equal to) the values specified in clause 8 of Rec. ITU-T
H.266:2020 | ISO/IEC 23090-3:2021.
For output timing conformance, it is a requirement that a conforming decoder shall also output the
cropped decoded pictures at the picture rates and times specified in Annex C of Rec. ITU-T H.266:2020
| ISO/IEC 23090-3:2021.
The display process, which ordinarily follows the output of the decoding process, is outside the scope of
this document.
6.5.4 Static tests for output order conformance
Static tests of a video decoder require testing of the samples of the cropped decoded pictures that
are output from the decoder, and can be accomplished when the decoded samples at the output of the
decoding process are available. It may not be possible to perform this type of test with a production
decoder (due to the lack of an appropriate accessible interface in the design at which to perform the test).
In such a case this test should be performed by the manufacturer during the design and development
phase. Static tests are used for testing the decoding process.
The pictures that are output by the decoder under test are checked to ensure that the following
requirements are fulfilled:
— The cropped decoded pictures that are output by the decoder under test shall correspond to those
that are output by the reference software decoder.
— The cropped decoded pictures that are output by the decoder under test shall be output in the same
order as those that are output by the reference software decoder.
— The values of the samples of the cropped decoded pictures that are output by the decoder under test
shall be identical to those that are output by the reference software decoder.
To assist with the checking of the decoding process and the cropped decoded pictures, hash values for
the cropped decoded pictures that are output by conforming decoders are provided in a corresponding
output picture log file for each test bitstream that is used in the specified conformance tests, and most
of these test bitstreams also contain decoded picture hash SEI messages that may be used for checking
the results of the decoding process of the decoder under test.
5
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/IEC 23090-15:2022(E)
6.5.5 Dynamic tests for output timing conformance
Dynamic tests are applied to check that all the decoded samples of the cropped decoded pictures are
output and that the timing of the output of the decoder's decoded samples conforms to the specifications
of Clause 8 and Annex C of Rec. ITU-T H.266:2020 | ISO/IEC 23090-3:2021, and to verify that the
decoder under test can operate according to bitstream flow characteristics prescribed by the specified
HRD models (as specified by the CPB and DPB specification in Annex C of Rec. ITU-T H.266:2020 |
ISO/IEC 23090-3:2021) when the bits of the bitstream are delivered at the proper rate.
The dynamic test is often easier to perform on a complete decoding system, which may include a
systems decoder, a video decoder and a display process. It may be possible to record the output of the
display process and to check that display order and timing of the cropped decoded pictures are correct
at the output of the display process. However, since the display process is not within the normative
scope of Rec. ITU-T H.266 | ISO/IEC 23090-3, there may be cases where the output of the display
process differs in timing or value even though the video decoder is conforming. In this case, the output
of the video decoder itself (before the display process) would need to be captured in order to perform
the dynamic tests on the video decoder. In particular the output order and timing of the output of the
cropped decoded pictures shall be correct.
If buffering period and picture timing SEI messages are included in the test bitstream, HRD conformance
shall be verified using the values of nal_initial_cpb_removal_delay, nal_initial_cpb_removal_offset, au_
cpb_removal_delay_minus1 and pic_dpb_output_delay that are included in the bitstream.
If buffering period and picture timing SEI messages are not included in the bitstream, the following
inferences shall be made to generate the missing parameters:
— fixed_pic_rate_general_flag[ i ] shall be inferred to be equal to 1.
— low_delay_hrd_flag[ i ] shall be inferred to be equal to 0.
— cbr_flag[ subLayerId ][ j ] shall be inferred to be equal to 0.
— The frame rate of the bitstream shall be inferred to be equal to the frame rate value specified in the
.txt file for the bitstream. If this is missing, then a frame rate of either 25 or 30000 ÷ 1001 can be
inferred.
— The bit rate of the bitstream shall be inferred to be equal to the maximum value for the level specified
in Table 136 in Rec. ITU-T H.266:2020 | ISO/IEC 23090-3:2021.
— CPB and DPB sizes shall be inferred to be equal to the maximum value for the level specified in
Table 135 in Rec. ITU-T H.266:2020 | ISO/IEC 23090-3:2021.
With the above inferences, the HRD shall be operated as follows:
— The CPB is filled starting at time t = 0, until it is full, before removal of the first access unit. This
means that the bp_nal_initial_cpb_removal_delay[ i ][ j ] shall be inferred to be equal to the total CPB
buffer size divided by the bit rate divided by 90000 (rounded downwards) and bp_vcl_initial_cpb_
removal_offset[ i ][ j ] shall be inferred to be equal to zero.
— The first access unit is removed at time t = bp_nal_initial_cpb_removal_delay[ i ][ j ] ÷ 90000 and
subsequent access units are removed at intervals based on the picture distance.
— Using these inferences with the accompanying bitstreams, the CPB will not overflow or underflow
and the DPB will not overflow.
6.5.6 Decoder conformance test for a particular profile, tier, and level
In order for a decoder for a particular profile, tier, and level to claim output order conformance to
Rec. ITU-T H.266 | ISO/IEC 23090-3, the decoder shall successfully pass the static test specified in
Clause 6.5.4 with all the bitstreams of the normative test suite specified for testing decoders of this
particular profile, tier, and level combination.
6
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 10 ----------------------
ISO/IEC 23090-15:2022(E)
In order for a decoder of a particular profile, tier, and level to claim output timing conformance to Rec.
ITU-T H.266 | ISO/IEC 23090-3, the decoder shall successfully pass both the static test specified in
clause 6.5.4 and the dynamic test specified in Clause 6.5.5 with all the bitstreams of the normative test
suite specified for testing decoders of this particular profile, tier, and level.
Tables 1 through 10 specify the normative test suites. The profile, tier, and level combinations are
described in the tables or in the .txt file associated with the bitstream.
6.6 Specification of the test bitstreams
6.6.1 General
Some characteristics of each bitstream listed in Table 1 are specified in this clause.
6.6.2 Test bitstreams – Coding tools for Main 10 profile with 4:2:0 chroma format and 10 bit
depth
6.6.2.1 Chroma separate tree (CST)
6.6.2.1.1 Test bitstream CST_A
Specification: All pictures are coded in I slices with CST enabled. CST is tested with all possible luma
and chroma block sizes, and luma-chroma block size combinations (e.g., luma block size is larger than,
equal to, or smaller than the corresponding chroma block size).
Functional stage: Reconstruction process.
Purpose: Check that the decoder can properly decode slices with CST enabled.
6.6.2.2 Dependent quantization (DQ)
6.6.2.2.1 Test bitstream DQ_A
Specification: The bitstream consists of three CVSs, with the following properties:
— The first CVS uses dependent quantization for all pictures, all non-related features (inter tools, ALF,
...) are disabled, and MTS and LFNST are disabled.
— The second CVS uses dependent quantization for all pictures, all non-related features (inter tools,
ALF, .) are disabled, and MTS (for intra) and LFNST are enabled.
— The third CVS exercises a picture-level selection between dependent quantization, sign data hiding,
and standard quantization, all non-related features (inter tools, ALF, .) are disabled, and MTS (for
intra) and LFNST are enabled.
Functional stage: Dependent quantization.
Purpose: Check that the decoder can properly decode slices with DQ enabled.
6.6.2.2.2 Test bitstream DQ_B
Specification: The bitstream consists of three CVSs of resolution 1920 x 1080, with the following
properties:
— The first CVS uses dependent quantization for all pictures, all non-related features (inter tools, ALF,
...) are disabled, and MTS and LFNST are disabled.
— The second CVS uses dependent quantization for all pictures, all non-related features (inter tools,
ALF, .) are disabled, and MTS (for intra) and LFNST are enabled.
7
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 11 ----------------------
ISO/IEC 23090-15:2022(E)
— The third CVS exercises a picture-level selection between dependent quantization, sign data hiding,
and standard quantization, all non-related features (inter tools, ALF, .) are disabled, and MTS (for
intra) and LFNST are enabled.
Functional stage: Dependent quantization.
Purpose: Check that the decoder can properly decode slices with DQ enabled.
6.6.2.3 Cross-component linear model (CCLM)
6.6.2.3.1 Test bitstream CCLM_A
Specification: The bitstream exercises corner cases for coding structures using CCLM with the
following properties:
— POC0: Chroma CU size is 64x64.
— POC1: First split of CU is horizontal, i.e. CU size is 64x32.
— POC2: First split of CU is quad, i.e. CU size is 32x32.
— POC3: First and second split of CU are horizontal and vertical, respectively.
— POC4: First split of CU is vertical or ternary, i.e. none of condition is satisfied for CCLM.
— POC5: CU size is 64x64 and ISP is enabled.
...