ISO/IEC 18181-3:2022
(Main)Information technology — JPEG XL Image Coding System — Part 3: Conformance testing
Information technology — JPEG XL Image Coding System — Part 3: Conformance testing
This document specifies the conformance testing of the ISO/IEC 18181 series, also known as JPEG XL. Other desirable aspects of implementation (including robustness and performance) are outside the scope of this document.
Titre manque — Partie 3: Titre manque
General Information
Relations
Standards Content (Sample)
INTERNATIONAL ISO/IEC
STANDARD 18181-3
First edition
2022-10
Information technology — JPEG XL
Image Coding System —
Part 3:
Conformance testing
Reference number
© ISO/IEC 2022
© ISO/IEC 2022
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii
© ISO/IEC 2022 – All rights reserved
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Testing procedure . 1
4.1 Structure of the test corpora . 1
4.1.1 General . 1
4.1.2 Numpy File Format . 2
4.2 Image similarity . 2
4.3 Conformance on a single test case . 3
4.4 Encoder test . 4
4.5 Decoder . 4
Annex A (informative) Description of the test corpora . 5
Bibliography . 7
iii
© ISO/IEC 2022 – All rights reserved
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance
are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria
needed for the different types of document should be noted. This document was drafted in
accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or
www.iec.ch/members_experts/refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC
list of patent declarations received (see https://patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see
www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 18181 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.
iv
© ISO/IEC 2022 – All rights reserved
Introduction
The ISO/IEC 18181 series, also known as "JPEG XL", supports lossless and lossy compression of images
and image sequences.
This document provides the framework, concepts and methodology for testing codestreams and
implementations, and the criteria to be achieved to claim conformance to the ISO/IEC 18181 series.
The objective of this standard is to promote interoperability between JPEG XL decoders, and to test
these systems for conformance. Conformance testing is the testing of a candidate implementation for
the existence of specific characteristics required by a standard.
The purpose of this document is to define a common test methodology, to provide a framework for
specific test suites, and to define the procedures to be followed during conformance testing.
v
© ISO/IEC 2022 – All rights reserved
INTERNATIONAL STANDARD ISO/IEC 18181-3:2022(E)
Information technology — JPEG XL Image Coding System —
Part 3:
Conformance testing
1 Scope
This document specifies the conformance testing of the ISO/IEC 18181 series, also known as JPEG XL.
Other desirable aspects of implementation (including robustness and performance) are outside the
scope of this document.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 18181-1, Information technology — JPEG XL image coding system — Part 1: Core coding system
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 18181-1 apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
4 Testing procedure
4.1 Structure of the test corpora
4.1.1 General
NOTE JPEG XL specifies infinite-precision arithmetic, whereas practical implementations are likely to
use finite-precision arithmetic. Allowing each intermediate operation to round the result to a floating-point
representation might not give sufficient freedom for alternate implementation strategies, or conversely too much
freedom to be useful. This document therefore specifies tolerances for the decoded samples.
A description of the test corpora is provided in Annex A and the following files are available from
https:// standards .iso .org/ iso -iec/ 18181/ -3/ ed -1/ en/ :
— a testcases/ subdirectory, containing
— multiple subdirectories, each of which contains a single test case for conformance testing
— multiple .txt files, each line of which contains the name of a folder corresponding to a test case
— a conformance.py file that provides a possible implementation of the testing procedure described in
this document
— a lcms2.py file, which is a supporting file for conformance.py
© ISO/IEC 2022 – All rights reserved
Each subdirectory contains:
[2]
— exactly one JSON file , named test.json
— exactly one JXL file, named input.jxl
— up to two NPY files, named reference_image.npy or reference_preview.npy
— up to two ICC profile files, named reference.icc or original.icc
— zero or one JPEG files, named reconstructed.jpeg
If a NPY file is present, then the ICC profile named reference.icc is also present.
Collectively, the NPY, ICC and JPEG files are known as the reference for this test case.
The JSON file specifies the tests to be performed as part of this test case.
The NPY files contain sample values for either the main frames or the preview frames, represented
as a four-dimensional array with dimensions (num_frames, height, width, num_channels) respectively.
num_frames is always 1 for the preview frames. The data format stored in these arrays is one of float (32
or 64 bits), uint8 and uint16.
“reference.icc” represents the colour space of the data in the NPY files.
“original.icc” represents the ICC profile of the samples that were used to create the JPEG XL file; this
is metadata that may be present in the JPEG XL file.
“reconstructed.jpg” contains the original JPEG1 file that produced a recompressed JPEG1 (i.e. with a
“jbrd” box) JPEG XL file.
4.1.2 Numpy File Format
A NPY file begins with the sequence of bytes “93 4e 55 4d 50 59” (byte 93 followed by “NUMPY” in
ASCII), followed by the sequence of bytes “01 00”, followed by two bytes representing a 16-bit little-
endian integer LEN that represents the length of the rest of the header.
The rest of the header contains the newline-terminated ASCII string “{'descr': '
order': False, 'shape': (X, Y, Z, W), }”, where X, Y, Z, W represent integers in their usual base-10
textual representation.
The NPY file represents an array with dimensions (X, Y, Z, W). The rest of the file contains X x Y x Z x
W little endian, 32-bit floating point numbers (as specified in IEEE 754-2019), in raster order with the
W dimension varying faster, i.e. floating point number in position (x, y, z, w) is stored in the 4 bytes
starting at position 4·(x·Y·Z·W + y·Z·W + z·W + w) after the header.
NOTE This is a subset of the format specified in
https:// numpy .org/ devdocs/ reference/ generated/ numpy .lib .format .html
4.2 Image similarity
Samples are compared in the colour space specified by the reference ICC profile “reference.icc”, with
nominal values in the interval [0, 1] (no clipping is to be applied to values outside this range). An image
is similar to the reference image if and only if the following three conditions are met:
1) The image dimensions, number of frames and number of channels are identical.
2) Each of their samples are similar, i.e. the peak error is bounded. A decoded sample D and reference
sample R are similar if and only if |D - R| is smaller or equal to a given threshold.
3) For all channels (components), the root mean square error (the root of the sum of (D - R) over all
samples) is smaller or equal to a given threshold.
© ISO/IEC 2022 – All rights reserved
For the purposes of this comparison, the channels are assumed to be ordered as follows: the first
three channels are the RGB channels (in the order R, G, B); then, channel 3+i is the i-th extra channel
(ISO/IEC 18181-1:2022, C.4.10).
In the case of greyscale images, the reference is given in RGB, where the RGB channels are equal (R = G
= B).
4.3 Conformance on a single test case
The JSON file for a given test case represents the configuration of the given test case.
The JSON file represents this configuration using a key-value format, where keys are strings and values
are strings, numbers, arrays or nested key-value pairs.
If the “frames” key is present, its corresponding value is an array of per-frame information. Its length
equals the number of frames in “reference_image.npy”. For each array entry, conformance is tested
by checking similarity of the i-th decoded frame with the i-th frame in “reference_image.npy”. For the
purposes of this document, the i-th decoded frame is defined to be the i-th frame with a FrameType
of either “kRegular” or “kSkipProgressive” and either a non-zero duration or is_last set to true, after
any blending and orientation is applied (ISO/IEC 18181-1:2022, C.1, C.2). This corresponds to frames
that are meant to be displayed to the end user. If the total number of decoded frames is not equal to
the length of the array, the decoder is non-conforming. The keys in the i-th entry of the array influence
conformance testing at follows:
— The key “rms_error” specifies the threshold for the root mean square error of any channel.
— The key “peak_error” specifies the threshold for the peak error o
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.