ISO/IEC 21122-4:2020
(Main)Information technology — JPEG XS low-latency lightweight image coding system — Part 4: Conformance testing
Information technology — JPEG XS low-latency lightweight image coding system — Part 4: Conformance testing
This document specifies the framework, concepts, methodology for testing, and criteria to be achieved to claim conformance to multiple parts of the ISO/IEC 21122 series. It lists the conformance testing procedures. This document specifies: — Conformance testing procedures for decoders implementing ISO/IEC 21122-1. — Conformance testing procedures for decoders implementing ISO/IEC 21122-3. — Tests to check codestreams for conformance to ISO/IEC 21122-1. As such, it provides means to test whether encoder implementations generate syntactically correct codestreams, and whether codestreams generated by such implementations follow the requirements of a particular profile, level and sublevel, and the buffer model implied by them. — Tests to check files for conformance to ISO/IEC 21122-3. — Conformance testing procedures that allow testing whether codestreams conform to any of the profiles specified in ISO/IEC 21122-2. — Conformance testing procedures that allow testing whether codestreams conform to the buffer model specified in ISO/IEC 21122-2 as part of a profile, level and sublevel. — Codestreams, decoded images, and error metrics to be used within the decoder testing procedures. — A buffer model test. — Abstract test suites. This document does not specify: — Testing the reconstruction of a full resolution image from a subsampled image format. In particular, upsampling from 4:2:2 to 4:4:4 sampling is a non-normative extension and as such its testing is beyond the scope of this document. — Testing the conversion of the sample values reconstructed by an ISO/IEC 21122-3 decoder to the target colour space by means of the colour specification box of ISO/IEC 21122-3. — Testing of the composition of background and foreground for images reconstructed from ISO/IEC 21122-3 files or codestreams. — Acceptance testing: the process of determining whether an implementation satisfies acceptance criteria and enables the user to determine whether or not to accept the implementation. This includes the planning and execution of several kinds of tests (e.g. functionality, quality, and speed performance testing) that demonstrate that the implementation satisfies the user requirements. — Performance testing: measures the performance characteristics of an implementation under test (IUT) such as its throughput, responsiveness, etc. under various conditions. — Robustness testing: the process of determining how well an implementation is able to conceal problems from attempting to reconstruct an image from an ill-formed codestream.
Technologies de l'information — Système de codage d'images léger à faible latence JPEG XS — Partie 4: Essais de conformité
General Information
Relations
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 21122-4
First edition
2020-06
Information technology — JPEG XS
low-latency lightweight image coding
system —
Part 4:
Conformance testing
Technologies de l'information — Système de codage d'images léger à
faible latence JPEG XS —
Partie 4: Essais de conformité
Reference number
ISO/IEC 21122-4:2020(E)
©
ISO/IEC 2020
---------------------- Page: 1 ----------------------
ISO/IEC 21122-4:2020(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2020
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2020 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 21122-4:2020(E)
Contents Page
Foreword .iv
1 Scope . 1
2 Normative references . 2
3 Terms and definitions . 2
4 Typesetting conventions . 3
5 Overview . 3
6 General description . 3
6.1 Overview . 3
6.2 Codestream syntax testing . 3
6.3 Test procedures to test decoders for conformance to ISO/IEC 21122-1 . 3
6.4 Test procedures to test decoders for conformance to ISO/IEC 21122-3 . 3
6.5 File format syntax testing . 3
6.6 Profile, level and sublevel conformance testing . 4
6.7 Buffer model conformance testing . 4
6.8 Electronic attachments . 4
7 Conformance files availability and updates . 4
Annex A (normative) Codestream syntax testing procedures . 5
Annex B (normative) Decoder testing procedures . 6
Annex C (normative) Decoder conformance tests . 9
Annex D (normative) File format syntax testing procedures .13
Annex E (normative) Buffer model conformance testing .14
Annex F (informative) Electronic attachments .16
Bibliography .18
© ISO/IEC 2020 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 21122-4:2020(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that
are members of ISO or IEC participate in the development of International Standards through
technical committees established by the respective organization to deal with particular fields of
technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other
international organizations, governmental and non-governmental, in liaison with ISO and IEC, also
take part in the work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC
list of patent declarations received (see http:// patents .iec .ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/
iso/ foreword .html.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 21122 series can be found on the ISO website.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/ members .html.
iv © ISO/IEC 2020 – All rights reserved
---------------------- Page: 4 ----------------------
INTERNATIONAL STANDARD ISO/IEC 21122-4:2020(E)
Information technology — JPEG XS low-latency lightweight
image coding system —
Part 4:
Conformance testing
1 Scope
This document specifies the framework, concepts, methodology for testing, and criteria to be achieved
to claim conformance to multiple parts of the ISO/IEC 21122 series. It lists the conformance testing
procedures.
This document specifies:
— Conformance testing procedures for decoders implementing ISO/IEC 21122-1.
— Conformance testing procedures for decoders implementing ISO/IEC 21122-3.
— Tests to check codestreams for conformance to ISO/IEC 21122-1. As such, it provides means to
test whether encoder implementations generate syntactically correct codestreams, and whether
codestreams generated by such implementations follow the requirements of a particular profile,
level and sublevel, and the buffer model implied by them.
— Tests to check files for conformance to ISO/IEC 21122-3.
— Conformance testing procedures that allow testing whether codestreams conform to any of the
profiles specified in ISO/IEC 21122-2.
— Conformance testing procedures that allow testing whether codestreams conform to the buffer
model specified in ISO/IEC 21122-2 as part of a profile, level and sublevel.
— Codestreams, decoded images, and error metrics to be used within the decoder testing procedures.
— A buffer model test.
— Abstract test suites.
This document does not specify:
— Testing the reconstruction of a full resolution image from a subsampled image format. In particular,
upsampling from 4:2:2 to 4:4:4 sampling is a non-normative extension and as such its testing is
beyond the scope of this document.
— Testing the conversion of the sample values reconstructed by an ISO/IEC 21122-3 decoder to the
target colour space by means of the colour specification box of ISO/IEC 21122-3.
— Testing of the composition of background and foreground for images reconstructed from
ISO/IEC 21122-3 files or codestreams.
— Acceptance testing: the process of determining whether an implementation satisfies acceptance
criteria and enables the user to determine whether or not to accept the implementation. This
includes the planning and execution of several kinds of tests (e.g. functionality, quality, and speed
performance testing) that demonstrate that the implementation satisfies the user requirements.
— Performance testing: measures the performance characteristics of an implementation under test
(IUT) such as its throughput, responsiveness, etc. under various conditions.
© ISO/IEC 2020 – All rights reserved 1
---------------------- Page: 5 ----------------------
ISO/IEC 21122-4:2020(E)
— Robustness testing: the process of determining how well an implementation is able to conceal
problems from attempting to reconstruct an image from an ill-formed codestream.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 21122-1, Information technology — JPEG XS low-latency lightweight image coding system —
Part 1: Core coding system
ISO/IEC 21122-2, Information technology — JPEG XS low-latency lightweight image coding system —
Part 2: Profiles and buffer models
ISO/IEC 21122-3, Information technology — JPEG XS low-latency lightweight image coding system —
Part 3: Transport and container formats
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 21122-3 and the
following apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
3.1
ASCII
American Standard Code for Information Interchange
binary encoding of 7-bit characters
Note 1 to entry: This encoding is as defined by ISO/IEC 646.
3.2
ETS
executable test suite
set of executable test cases in the form of codestreams that are input into an implementation under test
3.3
IUT
implementation under test
implementation that is being evaluated for conformance
3.4
JPEG XS file
sequence of bytes encoding an image in the JXS file format
Note 1 to entry: This format is specified in ISO/IEC 21122-3.
3.5
procedure
set of steps which accomplishes one of the tasks which comprise an encoding or decoding process
3.6
TCS
test codestream set
(sub-)set of codestreams that are input to an implementation under test for a particular test purpose
2 © ISO/IEC 2020 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC 21122-4:2020(E)
3.7
testing
process of evaluating conformance
4 Typesetting conventions
Regular face fonts as this text describe normative or informative text that provides specifications,
instructions, comments or details for the reader.
Monospaced text as this paragraph indicates file names, or program input or output as
necessary to either run software, or as generated by the software on the console.
5 Overview
The conformance files including codestreams, reference decoded images, descriptive files and auxiliary
software to facilitate testing are provided at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en, in
compressed form. File locations given in this document are expressed relative to the top level directory
tree within this compressed file. A Unix style file structure and delimiters are assumed.
This document contains instructions for the use of these files.
6 General description
6.1 Overview
The ISO/IEC 21122 series, also known as JPEG XS, consists of multiple parts of which for ISO/IEC 21122-1
describes the core coding system and the syntax of a codestream, for ISO/IEC 21122-2 describes
profiles and buffer models and for ISO/IEC 21122-3 describes transport and container formats. This
document defines test suites for decoder conformance tests for ISO/IEC 21122-1, ISO/IEC 21122-2 and
ISO/IEC 21122-3.
6.2 Codestream syntax testing
The procedures of Annex A shall be used for testing codestreams for syntactical correctness. They
depend on a codestream syntax parsing tool whose source code is provided at https:// standards .iso
.org/ iso -iec/ 21122/ -4/ ed -1/ en.
6.3 Test procedures to test decoders for conformance to ISO/IEC 21122-1
The procedures defined in Annex B and the subset of ETS defined in Annex C, subclauses C.2 to C.10
that correspond to the profiles, levels and sublevels supported by the decoder shall be used for testing
decoders for compliance to ISO/IEC 21122-1. These procedures and ETS allow an IUT to evaluate
conformance to ISO/IEC 21122-1 only.
6.4 Test procedures to test decoders for conformance to ISO/IEC 21122-3
The procedures defined in Annex B and the subset of ETS defined in Annex C, subclause C.11 that
correspond to the profiles, levels and sublevels supported by the decoder shall be used for testing
decoders for compliance to ISO/IEC 21122-3. These procedures and ETS allow an IUT to evaluate
conformance to ISO/IEC 21122-1 and ISO/IEC 21122-3.
6.5 File format syntax testing
The procedures that shall be used for testing JPEG XS files for compliance to the file format specified
in ISO/IEC 21221-3 are defined in Annex D. They depend on a codestream syntax parsing tool whose
source code is provided at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en.
© ISO/IEC 2020 – All rights reserved 3
---------------------- Page: 7 ----------------------
ISO/IEC 21122-4:2020(E)
6.6 Profile, level and sublevel conformance testing
Annex A also specifies a test procedure that shall be used for testing whether a codestream is
conforming to a particular profile, level and sublevel, and whether, in particular, its coding parameters
are within the constraints of the profile, level and sublevel indicated in the codestream. The test in
Annex A depends on a program that is given at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en.
6.7 Buffer model conformance testing
Annex E specifies test procedures that shall be used for testing whether a codestream is conforming
to a buffer model implied by the profile, level and sublevel indicated in the codestream. The tests in
Annex E depend on a program that is given at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en.
6.8 Electronic attachments
Annex F lists the electronic attachments to this document and describes how to compile and use them.
7 Conformance files availability and updates
The conformance test images, streams and conformance test software released with this document are
the latest tested versions available at the date at which the text was released.
4 © ISO/IEC 2020 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 21122-4:2020(E)
Annex A
(normative)
Codestream syntax testing procedures
A.1 General
This annex defines a procedure that shall be followed for determining whether a codestream is
syntactically well-formed and follows the syntactical requirements of ISO/IEC 21122-1. The test
procedure also checks whether the coding parameters of a given codestream are consistent with
the profile, level and sublevel indicators that are part of the picture header (see ISO/IEC 21122-1). A
typical use case for this test is to check whether a given encoder generates profile, level and sublevel
1)
information correctly. To this end, a Python test script jxscodestream.py that performs a syntax
analysis of a given codestream is provided at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en.
A.2 Installation
1
The test tool requires installation of a Python 2.7 interpreter on the computer system to be used for
performing a test. Python is available for multiple operating systems at https:// docs .python .org.
A.3 Usage of the syntax test tool
For testing a particular codestream for syntactical correctness and correct indication of profile, level
and sublevel, the syntax analyser jxscodestream.py tool shall be run on a command line as follows:
jxscodestream.py codestream.jxs
where codestream.jxs is the codestream to be tested. A codestream is not conforming in case the
above tool reports any error.
The lack of detection of any conformance violation by the syntax test tool should not be considered as a
definite proof that the codestream under testing conforms to all constraints required for conformance
2)
to ISO/IEC 21122-1 and ISO/IEC 21122-2. ISO/IEC 21122-5 provides additional means for testing a
codestream for conformance to the ISO/IEC 21122 series by feeding it into the reference software
implementation.
1) Python is a trademark of the Python Software Foundation. This information is given for the convenience of users
of this document and does not constitute an endorsement by ISO/IEC of the product named. Equivalent products
may be used if they can be shown to lead to the same results.
2) Under preparation. Stage at time of publication: ISO/IEC DIS 21122-5:2020.
© ISO/IEC 2020 – All rights reserved 5
---------------------- Page: 9 ----------------------
ISO/IEC 21122-4:2020(E)
Annex B
(normative)
Decoder testing procedures
B.1 General
This annex defines procedures that shall be followed for determining whether a JPEG XS decoder
implementation is conforming to a particular profile, level and sublevel.
B.2 Decoder test procedure
The following steps shall be performed for the purpose of testing a decoder implementation:
— Select to test for ISO/IEC 21122-1 only or ISO/IEC 21122-1 and ISO/IEC 21122-3.
— Select a profile, level and sublevel to test against.
— The profile, level and sublevel define a TCS (test codestream set), which consists of codestreams or
files within the test suite that conform to
— the codestream syntax specified in ISO/IEC 21122-1 or the file format specified in
ISO/IEC 21122-3,
— the selected profile,
— a level whose sample count and sample rate is smaller than or equal to the sample count and
sample rate of the selected level, and
— to a sublevel whose bitrate is smaller or equal than the bitrate of the selected sublevel.
— Decode each codestream or file using the implementation under test.
— The decoded outputs are format converted if necessary.
— The difference between the decoded outputs and reference outputs is measured.
— If the decoded and reference are identical, the test proceeds to the next codestream in the TCS.
— If the decoded image differs from the reference image, the implementation is non-conforming and
testing aborts.
The entire set of test codestreams is defined in more detail in a later clause of Annex C. Failure to match
the reference output for a single test codestream results in the decoder failing to be conforming with
the given profile, level and sublevel being tested.
B.3 Files for testing
A particular ETS defines the input codestreams. These are specified in Annex C for all profiles.
B.4 Decoder settings
Decoders may have mechanisms for supporting various decompression settings. These may be set in
the most advantageous way to achieve conformance. For example, a decoder with a "fast mode" and an
"accurate mode" may be set to the "accurate mode" to determine conformance. These settings should
6 © ISO/IEC 2020 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC 21122-4:2020(E)
be noted in any statement of conformance. Settings that allow the output resolution or spatial region of
the reference decoded images to
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.