Information technology — JPEG XS low-latency lightweight image coding system — Part 4: Conformance testing

This document specifies the framework, concepts, methodology for testing, and criteria to be achieved to claim conformance to multiple parts of the ISO/IEC 21122 series. It lists the conformance testing procedures. This document specifies: — Conformance testing procedures for decoders implementing ISO/IEC 21122-1. — Conformance testing procedures for decoders implementing ISO/IEC 21122-3. — Tests to check codestreams for conformance to ISO/IEC 21122-1. As such, it provides means to test whether encoder implementations generate syntactically correct codestreams, and whether codestreams generated by such implementations follow the requirements of a particular profile, level and sublevel, and the buffer model implied by them. — Tests to check files for conformance to ISO/IEC 21122-3. — Conformance testing procedures that allow testing whether codestreams conform to any of the profiles specified in ISO/IEC 21122-2. — Conformance testing procedures that allow testing whether codestreams conform to the buffer model specified in ISO/IEC 21122-2 as part of a profile, level and sublevel. — Codestreams, decoded images, and error metrics to be used within the decoder testing procedures. — A buffer model test. — Abstract test suites. This document does not specify: — Testing the reconstruction of a full resolution image from a subsampled image format. In particular, upsampling from 4:2:2 to 4:4:4 sampling is a non-normative extension and as such its testing is beyond the scope of this document. — Testing the conversion of the sample values reconstructed by an ISO/IEC 21122-3 decoder to the target colour space by means of the colour specification box of ISO/IEC 21122-3. — Testing of the composition of background and foreground for images reconstructed from ISO/IEC 21122-3 files or codestreams. — Acceptance testing: the process of determining whether an implementation satisfies acceptance criteria and enables the user to determine whether or not to accept the implementation. This includes the planning and execution of several kinds of tests (e.g. functionality, quality, and speed performance testing) that demonstrate that the implementation satisfies the user requirements. — Performance testing: measures the performance characteristics of an implementation under test (IUT) such as its throughput, responsiveness, etc. under various conditions. — Robustness testing: the process of determining how well an implementation is able to conceal problems from attempting to reconstruct an image from an ill-formed codestream.

Technologies de l'information — Système de codage d'images léger à faible latence JPEG XS — Partie 4: Essais de conformité

General Information

Status
Published
Publication Date
09-Jun-2020
Current Stage
9599 - Withdrawal of International Standard
Completion Date
05-Oct-2022
Ref Project

Buy Standard

Standard
ISO/IEC 21122-4:2020 - Information technology -- JPEG XS low-latency lightweight image coding system
English language
18 pages
sale 15% off
Preview
sale 15% off
Preview

Standards Content (sample)

INTERNATIONAL ISO/IEC
STANDARD 21122-4
First edition
2020-06
Information technology — JPEG XS
low-latency lightweight image coding
system —
Part 4:
Conformance testing
Technologies de l'information — Système de codage d'images léger à
faible latence JPEG XS —
Partie 4: Essais de conformité
Reference number
ISO/IEC 21122-4:2020(E)
ISO/IEC 2020
---------------------- Page: 1 ----------------------
ISO/IEC 21122-4:2020(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2020

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2020 – All rights reserved
---------------------- Page: 2 ----------------------
ISO/IEC 21122-4:2020(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 2

3 Terms and definitions ..................................................................................................................................................................................... 2

4 Typesetting conventions .............................................................................................................................................................................. 3

5 Overview ....................................................................................................................................................................................................................... 3

6 General description ........................................................................................................................................................................................... 3

6.1 Overview ...................................................................................................................................................................................................... 3

6.2 Codestream syntax testing ............................................................................................................................................................ 3

6.3 Test procedures to test decoders for conformance to ISO/IEC 21122-1 .............................................. 3

6.4 Test procedures to test decoders for conformance to ISO/IEC 21122-3 .............................................. 3

6.5 File format syntax testing .............................................................................................................................................................. 3

6.6 Profile, level and sublevel conformance testing ......................................................................................................... 4

6.7 Buffer model conformance testing ........................................................................................................................................ 4

6.8 Electronic attachments .................................................................................................................................................................... 4

7 Conformance files availability and updates ............................................................................................................................. 4

Annex A (normative) Codestream syntax testing procedures ................................................................................................... 5

Annex B (normative) Decoder testing procedures ................................................................................................................................ 6

Annex C (normative) Decoder conformance tests .................................................................................................................................. 9

Annex D (normative) File format syntax testing procedures ..................................................................................................13

Annex E (normative) Buffer model conformance testing ............................................................................................................14

Annex F (informative) Electronic attachments .......................................................................................................................................16

Bibliography .............................................................................................................................................................................................................................18

© ISO/IEC 2020 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO/IEC 21122-4:2020(E)
Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical

Commission) form the specialized system for worldwide standardization. National bodies that

are members of ISO or IEC participate in the development of International Standards through

technical committees established by the respective organization to deal with particular fields of

technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other

international organizations, governmental and non-governmental, in liaison with ISO and IEC, also

take part in the work.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for

the different types of document should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject

of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent

rights. Details of any patent rights identified during the development of the document will be in the

Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents) or the IEC

list of patent declarations received (see http:// patents .iec .ch).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso .org/

iso/ foreword .html.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,

Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.

A list of all parts in the ISO/IEC 21122 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A

complete listing of these bodies can be found at www .iso .org/ members .html.
iv © ISO/IEC 2020 – All rights reserved
---------------------- Page: 4 ----------------------
INTERNATIONAL STANDARD ISO/IEC 21122-4:2020(E)
Information technology — JPEG XS low-latency lightweight
image coding system —
Part 4:
Conformance testing
1 Scope

This document specifies the framework, concepts, methodology for testing, and criteria to be achieved

to claim conformance to multiple parts of the ISO/IEC 21122 series. It lists the conformance testing

procedures.
This document specifies:
— Conformance testing procedures for decoders implementing ISO/IEC 21122-1.
— Conformance testing procedures for decoders implementing ISO/IEC 21122-3.

— Tests to check codestreams for conformance to ISO/IEC 21122-1. As such, it provides means to

test whether encoder implementations generate syntactically correct codestreams, and whether

codestreams generated by such implementations follow the requirements of a particular profile,

level and sublevel, and the buffer model implied by them.
— Tests to check files for conformance to ISO/IEC 21122-3.

— Conformance testing procedures that allow testing whether codestreams conform to any of the

profiles specified in ISO/IEC 21122-2.

— Conformance testing procedures that allow testing whether codestreams conform to the buffer

model specified in ISO/IEC 21122-2 as part of a profile, level and sublevel.

— Codestreams, decoded images, and error metrics to be used within the decoder testing procedures.

— A buffer model test.
— Abstract test suites.
This document does not specify:

— Testing the reconstruction of a full resolution image from a subsampled image format. In particular,

upsampling from 4:2:2 to 4:4:4 sampling is a non-normative extension and as such its testing is

beyond the scope of this document.

— Testing the conversion of the sample values reconstructed by an ISO/IEC 21122-3 decoder to the

target colour space by means of the colour specification box of ISO/IEC 21122-3.

— Testing of the composition of background and foreground for images reconstructed from

ISO/IEC 21122-3 files or codestreams.

— Acceptance testing: the process of determining whether an implementation satisfies acceptance

criteria and enables the user to determine whether or not to accept the implementation. This

includes the planning and execution of several kinds of tests (e.g. functionality, quality, and speed

performance testing) that demonstrate that the implementation satisfies the user requirements.

— Performance testing: measures the performance characteristics of an implementation under test

(IUT) such as its throughput, responsiveness, etc. under various conditions.
© ISO/IEC 2020 – All rights reserved 1
---------------------- Page: 5 ----------------------
ISO/IEC 21122-4:2020(E)

— Robustness testing: the process of determining how well an implementation is able to conceal

problems from attempting to reconstruct an image from an ill-formed codestream.
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO/IEC 21122-1, Information technology — JPEG XS low-latency lightweight image coding system —

Part 1: Core coding system

ISO/IEC 21122-2, Information technology — JPEG XS low-latency lightweight image coding system —

Part 2: Profiles and buffer models

ISO/IEC 21122-3, Information technology — JPEG XS low-latency lightweight image coding system —

Part 3: Transport and container formats
3 Terms and definitions

For the purposes of this document, the terms and definitions given in ISO/IEC 21122-3 and the

following apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at http:// www .electropedia .org/
3.1
ASCII
American Standard Code for Information Interchange
binary encoding of 7-bit characters
Note 1 to entry: This encoding is as defined by ISO/IEC 646.
3.2
ETS
executable test suite

set of executable test cases in the form of codestreams that are input into an implementation under test

3.3
IUT
implementation under test
implementation that is being evaluated for conformance
3.4
JPEG XS file
sequence of bytes encoding an image in the JXS file format
Note 1 to entry: This format is specified in ISO/IEC 21122-3.
3.5
procedure

set of steps which accomplishes one of the tasks which comprise an encoding or decoding process

3.6
TCS
test codestream set

(sub-)set of codestreams that are input to an implementation under test for a particular test purpose

2 © ISO/IEC 2020 – All rights reserved
---------------------- Page: 6 ----------------------
ISO/IEC 21122-4:2020(E)
3.7
testing
process of evaluating conformance
4 Typesetting conventions

Regular face fonts as this text describe normative or informative text that provides specifications,

instructions, comments or details for the reader.

Monospaced text as this paragraph indicates file names, or program input or output as

necessary to either run software, or as generated by the software on the console.

5 Overview

The conformance files including codestreams, reference decoded images, descriptive files and auxiliary

software to facilitate testing are provided at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en, in

compressed form. File locations given in this document are expressed relative to the top level directory

tree within this compressed file. A Unix style file structure and delimiters are assumed.

This document contains instructions for the use of these files.
6 General description
6.1 Overview

The ISO/IEC 21122 series, also known as JPEG XS, consists of multiple parts of which for ISO/IEC 21122-1

describes the core coding system and the syntax of a codestream, for ISO/IEC 21122-2 describes

profiles and buffer models and for ISO/IEC 21122-3 describes transport and container formats. This

document defines test suites for decoder conformance tests for ISO/IEC 21122-1, ISO/IEC 21122-2 and

ISO/IEC 21122-3.
6.2 Codestream syntax testing

The procedures of Annex A shall be used for testing codestreams for syntactical correctness. They

depend on a codestream syntax parsing tool whose source code is provided at https:// standards .iso

.org/ iso -iec/ 21122/ -4/ ed -1/ en.
6.3 Test procedures to test decoders for conformance to ISO/IEC 21122-1

The procedures defined in Annex B and the subset of ETS defined in Annex C, subclauses C.2 to C.10

that correspond to the profiles, levels and sublevels supported by the decoder shall be used for testing

decoders for compliance to ISO/IEC 21122-1. These procedures and ETS allow an IUT to evaluate

conformance to ISO/IEC 21122-1 only.
6.4 Test procedures to test decoders for conformance to ISO/IEC 21122-3

The procedures defined in Annex B and the subset of ETS defined in Annex C, subclause C.11 that

correspond to the profiles, levels and sublevels supported by the decoder shall be used for testing

decoders for compliance to ISO/IEC 21122-3. These procedures and ETS allow an IUT to evaluate

conformance to ISO/IEC 21122-1 and ISO/IEC 21122-3.
6.5 File format syntax testing

The procedures that shall be used for testing JPEG XS files for compliance to the file format specified

in ISO/IEC 21221-3 are defined in Annex D. They depend on a codestream syntax parsing tool whose

source code is provided at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en.

© ISO/IEC 2020 – All rights reserved 3
---------------------- Page: 7 ----------------------
ISO/IEC 21122-4:2020(E)
6.6 Profile, level and sublevel conformance testing

Annex A also specifies a test procedure that shall be used for testing whether a codestream is

conforming to a particular profile, level and sublevel, and whether, in particular, its coding parameters

are within the constraints of the profile, level and sublevel indicated in the codestream. The test in

Annex A depends on a program that is given at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en.

6.7 Buffer model conformance testing

Annex E specifies test procedures that shall be used for testing whether a codestream is conforming

to a buffer model implied by the profile, level and sublevel indicated in the codestream. The tests in

Annex E depend on a program that is given at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en.

6.8 Electronic attachments

Annex F lists the electronic attachments to this document and describes how to compile and use them.

7 Conformance files availability and updates

The conformance test images, streams and conformance test software released with this document are

the latest tested versions available at the date at which the text was released.
4 © ISO/IEC 2020 – All rights reserved
---------------------- Page: 8 ----------------------
ISO/IEC 21122-4:2020(E)
Annex A
(normative)
Codestream syntax testing procedures
A.1 General

This annex defines a procedure that shall be followed for determining whether a codestream is

syntactically well-formed and follows the syntactical requirements of ISO/IEC 21122-1. The test

procedure also checks whether the coding parameters of a given codestream are consistent with

the profile, level and sublevel indicators that are part of the picture header (see ISO/IEC 21122-1). A

typical use case for this test is to check whether a given encoder generates profile, level and sublevel

information correctly. To this end, a Python test script jxscodestream.py that performs a syntax

analysis of a given codestream is provided at https:// standards .iso .org/ iso -iec/ 21122/ -4/ ed -1/ en.

A.2 Installation

The test tool requires installation of a Python 2.7 interpreter on the computer system to be used for

performing a test. Python is available for multiple operating systems at https:// docs .python .org.

A.3 Usage of the syntax test tool

For testing a particular codestream for syntactical correctness and correct indication of profile, level

and sublevel, the syntax analyser jxscodestream.py tool shall be run on a command line as follows:

jxscodestream.py codestream.jxs

where codestream.jxs is the codestream to be tested. A codestream is not conforming in case the

above tool reports any error.

The lack of detection of any conformance violation by the syntax test tool should not be considered as a

definite proof that the codestream under testing conforms to all constraints required for conformance

to ISO/IEC 21122-1 and ISO/IEC 21122-2. ISO/IEC 21122-5 provides additional means for testing a

codestream for conformance to the ISO/IEC 21122 series by feeding it into the reference software

implementation.

1) Python is a trademark of the Python Software Foundation. This information is given for the convenience of users

of this document and does not constitute an endorsement by ISO/IEC of the product named. Equivalent products

may be used if they can be shown to lead to the same results.
2) Under preparation. Stage at time of publication: ISO/IEC DIS 21122-5:2020.
© ISO/IEC 2020 – All rights reserved 5
---------------------- Page: 9 ----------------------
ISO/IEC 21122-4:2020(E)
Annex B
(normative)
Decoder testing procedures
B.1 General

This annex defines procedures that shall be followed for determining whether a JPEG XS decoder

implementation is conforming to a particular profile, level and sublevel.
B.2 Decoder test procedure

The following steps shall be performed for the purpose of testing a decoder implementation:

— Select to test for ISO/IEC 21122-1 only or ISO/IEC 21122-1 and ISO/IEC 21122-3.

— Select a profile, level and sublevel to test against.

— The profile, level and sublevel define a TCS (test codestream set), which consists of codestreams or

files within the test suite that conform to

— the codestream syntax specified in ISO/IEC 21122-1 or the file format specified in

ISO/IEC 21122-3,
— the selected profile,

— a level whose sample count and sample rate is smaller than or equal to the sample count and

sample rate of the selected level, and

— to a sublevel whose bitrate is smaller or equal than the bitrate of the selected sublevel.

— Decode each codestream or file using the implementation under test.
— The decoded outputs are format converted if necessary.
— The difference between the decoded outputs and reference outputs is measured.

— If the decoded and reference are identical, the test proceeds to the next codestream in the TCS.

— If the decoded image differs from the reference image, the implementation is non-conforming and

testing aborts.

The entire set of test codestreams is defined in more detail in a later clause of Annex C. Failure to match

the reference output for a single test codestream results in the decoder failing to be conforming with

the given profile, level and sublevel being tested.
B.3 Files for testing

A particular ETS defines the input codestreams. These are specified in Annex C for all profiles.

B.4 Decoder settings

Decoders may have mechanisms for supporting various decompression settings. These may be set in

the most advantageous way to achieve conformance. For example, a decoder with a "fast mode" and an

"accurate mode" may be set to the "accurate mode" to determine conformance. These settings should

6 © ISO/IEC 2020 – All rights reserved
---------------------- Page: 10 ----------------------
ISO/IEC 21122-4:2020(E)

be noted in any statement of conformance. Settings that allow the output resolution or spatial region of

the reference decoded images to
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.