ISO/IEC 29170-2:2015

Information technology - Advanced image coding and evaluation - Part 2: Evaluation procedure for nearly lossless coding

ISO/IEC 29170-2:2015 normalizes evaluation and grading of a light coding system used for displays and display systems, but is independent of the display technology. This procedure measures whether an observer can distinguish between an uncompressed reference and the reconstructed image to a pre-determined, statistically meaningful level. The procedure compares individual images with two possible forced choice comparison test methods. This procedure relies on subjective evaluation methods designed to discern image imperfections on electronic colour displays of any technology or size.

Technologies de l'information — Codage d'image avancé et évaluation — Partie 2: Mode opératoire d'évaluation pour codage presque sans perte

General Information

• Status: Published
• Publication Date: 12-Aug-2015
• ICS: 35.040 - Information coding
• ICS: 35.040.30 - Coding of graphical and photographical information
• Technical Committee: ISO/IEC JTC 1/SC 29 - Coding of audio, picture, multimedia and hypermedia information
• Drafting Committee: ISO/IEC JTC 1/SC 29/WG 1 - JPEG Coding of digital representations of images
• Current Stage: 9093 - International Standard confirmed
• Start Date: 24-Jun-2021
• Completion Date: 30-Oct-2025

Relations

• Amended By: ISO/IEC 29170-2:2015/Amd 1:2020 - Information technology - Advanced image coding and evaluation - Part 2: Evaluation procedure for nearly lossless coding - Amendment 1: Evaluation procedure parameters for nearly lossless coding of high dynamic range media and image sequences
• Effective Date: 01-Jan-2022

Overview

ISO/IEC 29170-2:2015 - "Information technology - Advanced image coding and evaluation - Part 2: Evaluation procedure for nearly lossless coding" standardizes a subjective test procedure to determine whether light (nearly lossless) image coding used in display streams is visually lossless. The procedure is display-technology independent and measures if observers can distinguish a reconstructed (coded) image from an uncompressed reference to a statistically meaningful level using forced-choice psychophysical tests.

Key Topics

• Test protocols: Two forced-choice methods are defined - a non-flickering paired comparison (Annex A) and a more sensitive interleaved-image forced-choice test (Annex B).
• Observer selection & screening: Observers are drawn from general population (experts and non-experts); recommended age range 18–30 years and not exceeding 40. Visual acuity and colour vision checks are required (Snellen 20/20 at 50 cm; Ishihara test). Corrective lenses with multiple focal lengths are disallowed.
• Media selection: Procedures for choosing representative image content appropriate to the target display and use-case are provided (Annex C).
• Viewing conditions: Requirements cover display calibration, lighting, viewing distance, viewing position and viewing time to ensure reproducible subjective testing.
• Test execution: Training sessions, trial structure, retries, block/session definitions, and minimum observer counts (at least 10 passing observers recommended) are specified.
• Evaluation metrics: The procedure uses an Acceptable Quality Level (AQL) and just-noticeable-difference concepts to quantify visually lossless performance.
• Reporting & verification: Test reporting templates and self-test certification guidance are included (Annex D, Annex F).

Applications and Who Uses It

ISO/IEC 29170-2:2015 is intended for:

• Display manufacturers (TV, monitor, mobile panels) validating visually lossless compression on display links.
• Codec and encoder developers working on low-latency, low-complexity compression for display streams.
• System integrators and SoC/graphics engineers evaluating trade-offs between compression, bandwidth, and power.
• Quality assurance and conformance labs performing subjective acceptance testing.
• Standards bodies and researchers comparing coding algorithms under controlled subjective conditions.

Practical uses include validating display stream compression for set-top boxes, mobile device display interfaces, GPU-to-panel links, and any application where compressed streams must be perceptually indistinguishable from originals.

Related Standards

• ISO 3664 (viewing conditions)
• ISO 20462-2 (image evaluation practices)
• ISO/IEC TR 29170-1 (background and guidelines)

Keywords: ISO/IEC 29170-2, visually lossless, nearly lossless coding, display stream compression, forced-choice test, subjective evaluation, AQL, observer screening.

Standards Content (Sample)

INTERNATIONAL ISO/IEC
STANDARD 29170-2
First edition
2015-08-15
Information technology — Advanced
image coding and evaluation —
Part 2:
Evaluation procedure for nearly
lossless coding
Technologies de l’information — Codage d’image avancé et
évaluation —
Partie 2: Mode opératoire d’évaluation pour codage presque sans
perte
Reference number
©
ISO/IEC 2015
© ISO/IEC 2015, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. [phone]
Fax [phone]
[e-mail]
www.iso.org
ii © ISO/IEC 2015 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Symbols and abbreviated terms . 3
5 Test methods . 3
5.1 Protocol selection . 3
5.2 Media selection . 3
5.3 Observers . 3
5.3.1 Observer selection . 3
5.3.2 Observer visual screening. 4
5.3.3 Instructions to observers . 4
5.3.4 Training session . 4
5.4 Viewing conditions . 5
5.4.1 Lighting and display calibration . 5
5.4.2 Viewing distance . 5
5.4.3 Viewing position . 6
5.5 Viewing time . 7
5.6 Trial retry . 7
5.7 Test reporting . 7
5.8 Verification of procedure . 7
Annex A (normative) Forced choice paradigm with non-flickering images test protocol .8
Annex B (normative) Forced choice paradigm with interleaved images test protocol .10
Annex C (normative) Media selection and preparation procedure .12
Annex D (normative) Test report procedure .15
Annex E (informative) Observer vision testing .21
Annex F (informative) Self-test certification .22
Bibliography .25
© ISO/IEC 2015 – All rights reserved iii

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT), see the following URL: Foreword — Supplementary information.
The committee responsible for this document is ISO/IEC JTC 1, Information technology, Subcommittee
SC 29, Coding of audio, picture, multimedia and hypermedia information.
ISO/IEC 29170 consists of the following parts, under the general title Information technology —
Advanced image coding and evaluation:
— Part 1: Guidelines for coding system evaluation (forthcoming)
— Part 2: Evaluation procedure for visually lossless coding
iv © ISO/IEC 2015 – All rights reserved

Introduction
This International Standard normalizes a procedure to evaluate coding systems by subjective methods.
The procedure is particularly useful for evaluating lightly compressed coding systems used, for instance,
in display stream compression where a source compresses image data sent to a display. Examples of
display streams include but are not limited to a wired link between a set-top box unit and a television
or between a mobile host graphics processor and a display panel module in a mobile appliance. Viewers
of these displays should be unaware that a coding system is employed in the device or system. A coding
system will be considered visually lossless if the test results meet a pre-defined acceptable quality level
demonstrated by the performance criteria described in this Specification under the viewing conditions
specified and media sets provided.
Appliances that may require visually lossless performance for compressed display streams include:
computer monitor displays, televisions, mobile phone and tablet displays. Data compression for these
systems allows existing display links to carry more display data than is possible with uncompressed
image streams or to reduce system power consumption or both.
The types of coding systems tested by this procedure may have the following properties:
a) The presence of a coding system should be undetectable to a user who is viewing the display.
b) The coding system operates in real-time, with negligible latency, low complexity hardware and
minimal memory in both the encoder and the decoder.
This procedure builds on prior standardization and best practices embodied in ISO 3664, ISO 20462-2
and ISO/IEC TR 29170-1.
© ISO/IEC 2015 – All rights reserved v

INTERNATIONAL STANDARD ISO/IEC 29170-2:2015(E)
Information technology — Advanced image coding and
evaluation —
Part 2:
Evaluation procedure for nearly lossless coding
1 Scope
This Technical Specification normalizes evaluation and grading of a light coding system used for
displays and display systems, but is independent of the display technology. This procedure measures
whether an observer can distinguish between an uncompressed reference and the reconstructed image
to a pre-determined, statistically meaningful level.
The procedure compares individual images with two possible forced choice comparison test methods.
This procedure relies on subjective evaluation methods designed to discern image imperfections on
electronic colour displays of any technology or size.
Image selections for testing a specific coding system has bearing on the results this procedure will
yield, but specific images required for testing are not within scope, excluding an informative annex
describing self-test certification. Image categories may vary between end-usage products. For
example, content relevant to television manufacturers may or may not be relevant to computer display
manufacturers. Due to the nature of this procedure as a visual psychophysical test, observer’s age is
considered a meaningful parameter of the results.
2 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies.
ISO 3664, Graphic technology and photography — Viewing conditions
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
3.1
advance time
time a reference image or test image is displayed during an interleaved pair comparison test
3.2
algorithm
unique combination of test conditions that contribute to a unique test image, for example, the
combination of coding an image with one compression level and one coding method represents coding
with one algorithm
Note 1 to entry: Coding an image with a different compression level and the same coding method represents a
second unique algorithm.
3.3
blank time
time between trials when the display shows no stimulus
© ISO/IEC 2015 – All rights reserved 1

3.4
block of trials
long experiment is logically divided into a series of trials
Note 1 to entry: There is an expectation that the observer iteratively completes each trial presented during a
short block of time without stopping for breaks beyond those allocated by the software presentation timing.
3.5
control image
cropped test image coded to ensure defects are easily detectable in a paired comparison test
3.6
evaluator
expert in the field of image or compression artefact analysis or vision science, who prepares images,
categorizes images or instructs the observer
3.7
expert observer
observer skilled in vision science or coding technology and image artefacts
3.8
image
still representation of pixels rendered by the display
3.9
just noticeable difference
stimulus difference that would lead to a 0,75 probability of correct responses in a two-alternative
forced-choice task
3.10
non-expert observer
observer with no special skills or training in vision science or coding technology
3.11
observer
individual performing the subjective evaluation procedure by evaluating stimulus to make paired
comparison task choices
3.12
picture element
pixel
smallest element that is capable of generating the full intended functionality (e.g. colour and gray scale)
of the display
Note 1 to entry: In a multicolour display, the smallest addressable element capable of producing the full colour
range or the smallest element that is capable of generating the full functionality of the display.
3.13
reference image
cropped original image displayed during a trial
3.14
session
description of the full time interval in which an observer participates in the experiment on a given day
Note 1 to entry: It can encompass multiple blocks of trials. A full experiment can be divided into multiple sessions.
3.15
test image
cropped version of a reconstructed image after coding displayed during a trial
2 © ISO/IEC 2015 – All rights reserved

3.16
trial
single unit of the experiment
Note 1 to entry: A set of images will be presented on the screen for a defined viewing time. The trial is complete
only when the observer has entered a response.
3.17
viewing time
number of seconds an observer views paired stimuli. The viewing time has a maximum value in all
situations
3.18
visually lossless
fully decoded image or multimedia sample presenting a display that is visually indistinguishable from
the original uncompressed data over the same spatial area and when viewed under the same conditions
4 Symbols and abbreviated terms
AQL acceptable quality level
PPD pixels per degree
RGB red-green-blue
5 Test methods
5.1 Protocol selection
The evaluator shall select the most suitable test protocol from those listed in Table 1 and contained in
the listed annex. The protocol selection depends on the media under test, end display usage and the
target AQL.
Table 1 — Visually lossless testing protocol
Test Protocol Comment
Annex A forced choice paradigm with a non-flick- sensitive testing, suitable for testing images compressed
ering image and reconstructed by light coding systems, but not as
sensitive as interleaved testing
Annex B forced choice paradigm with interleaved most sensitive testing, suitable for testing images com-
images pressed and reconstructed by light coding systems
5.2 Media selection
Media selection takes images or other media for display rendering relevant to the display type and user
applications.
Evaluators shall follow media selection procedures in Annex C after selecting a test protocol.
5.3 Observers
5.3.1 Observer selection
The observers shall be selected from a general population that may include both experts and non-
experts. The observers for the experiment shall not include evaluators who participated in the media
selection for the experiment being conducted.
© ISO/IEC 2015 – All rights reserved 3

The observer population should include variations in gender, ethnicity and age. The experiment is visual
in nature and age can strongly correlate to visual acuity, therefore, participant age for this procedure
favours the age range for the observer from 18 to 30 years old.
An observer’s age shall not exceed 40 years.
This procedure recommends evaluators recruit a suitable number of observers sufficient to include no
less than 10 observers who pass visual acuity (see 5.3.2) and test reporting (see D.1.2) requirements of
this Technical Specification.
5.3.2 Observer visual screening
The following selection criteria shall apply:
a) Observers may wear corrective lens, either glasses or contacts that shall not have multiple focal
lengths, e.g. progressive, bifocal or trifocal corrective lens.
b) Observers shall demonstrate normal visual acuity verified by using a Snellen reading test chart
where the observer reads at 20/20 from 50 cm.
c) Participants shall demonstrate normal colour vision verified by testing with Ishihara plates or
equivalent.
Evaluators can refer to Annex E for tools that help assess an observer’s visual acuity.
5.3.3 Instructions to observers
Evaluators shall provide equivalent instructions to all observers with the criteria listed, here:
a) Explain the use of the software to record image assessments.
b) Explain the forced choice task. Every trial requires a response, even a guess.
c) Explain what to do if a conscious mistake is made, such as a finger slip, or the observer looked away
and did not see the stimuli. If a retry is permitted (see 5.6), explain the method to retry that trial.
d) Explain where to sit and how to arrange the chair and use of the chin and headrest for the proper
viewing position and comfort.
5.3.4 Training session
The evaluator should use a training session consisting of one block of trials for observers new to
software used in subjective image testing. A training session will:
a) Use the control images from the experiment as test images.
b) Do not report data from the training session in the data results.
c) Use the same viewing time limit as the experiment.
d) Prompt the observer when a correct or incorrect response is entered. If incorrect, continue by
repeating the test image until a correct response is entered.
Observers who have participated in the experiment usually do not need to repeat a training session.
4 © ISO/IEC 2015 – All rights reserved

5.4 Viewing conditions
5.4.1 Lighting and display calibration
Viewing conditions shall be consistent with ISO 3664 viewing condition for images displayed on colour
monitors. Exceptions include:
2 2
a) The luminance of the white displayed on the monitor shall be between 100 cd/m and 140 cd/m .
b) Displays that do not contain calibration tables should be avoided, but if used should have gamma
setting verified close to 2,2 and deviations noted in the test report. Record maximum luminance of
the display used in the procedure.
c) The colour of surrounding walls and ceilings do not require a specific colour but shall not cause
distracting reflections that may affect the vision of the observer. An appropriate viewing booth is
optionally desirable.
d) The display monitor shall render at least the number of bits per component present in the tested
images.
5.4.2 Viewing distance
Viewing distance shall be controlled and ensured using a chin and forehead rest for observers. Figure 1
shows an example chin and forehead rest. The rest may be height adjustable to ensure small differences
between observer’s head size can be accommodated, the chair for the observer should have a large
adjustment range to ensure the observer is as comfortable as possible during the procedure.
Figure 1 — Example chin and forehead rest
© ISO/IEC 2015 – All rights reserved 5

The viewing distance is a function of the pixels per degree (PPD) to be subtended at the set viewing
distance.
The viewing distance shall be determined according to Table 2 based on horizontal pixels, the display
pixel density and pixels subtended in one degree of arc.
Table 2 — Viewing distance versus display size and resolution
a b
Condition PPD D
cm
D equals the larger of the value in
c
the following equation or 12cm
W
Viewing distance 30
D=
H ×tan( )
RES
PPD
a
The experiment requires a consistent display orientation to be maintained and mobile display may have a different
width and pixel resolution in landscape versus portrait orientation. PPD is calculated for each orientation. Detailed work
on computer displays and mobile devices tends to be closer than for general entertainment, e.g., television, and requires
evaluation with a more aggressive PPD than would be the case for Snellen acuity (30 cycles/degree or PPD = 60)
b
W is the screen width (cm) and H is the number of pixels across the display horizontally as viewed by the
RES
observer.
c
The minimum focusing distance for normal vision is predetermined as 12 cm by this Technical Specification.
5.4.3 Viewing position
An observer shall be seated in a comfortable position oriented with respect to the display as shown in
Figure 2 and Figure 3:
Figure 2 — Observer position with respect to the test display (side view)
6 © ISO/IEC 2015 – All rights reserved

Figure 3 — Observer position with respect to the test display (top view)
5.5 Viewing time
An observer shall view paired stimuli on the display for no more than 4 s.
If no choice is made within the viewing time, the display shall blank out the stimuli and prompt the
observer for a response. The blanking of the screen should use a dark, neutral colour with low luminance
in order to not hinder viewing the next stimulus pair. User prompts from the screen should be viewable
with mesopic vision, preferable using red lettering.
If the experiment permits retries (see 5.6) of a paired stimulus, the viewing time is reset for each
viewing attempt.
5.6 Trial retry
An experiment may permit an observer to retry a trial. The evaluator provides a mechanism to retry
paired stimuli when the observer makes a conscious mistake or does not see the trial stimuli. The retry
data replaces the prior recorded response as if it did not occur. Evaluators should track the number of
retries by each observer over the course of all sessions.
5.7 Test reporting
Evaluators shall report results of the data collected by the procedure as prescribed in Annex D.
5.8 Verification of procedure
Evaluators may self-certify a specific implementation of this procedure by correlating a trial to the
results shown in Annex F.
© ISO/IEC 2015 – All rights reserved 7

Annex A
(normative)
Forced choice paradigm with non-flickering images test protocol
A.1 Image pair generation
A.1.1 On-screen presentation of images
Each test image shall be presented side-by-side with the reference image. These images shall be
separated by a 1,0° ± 0,1° gap separating them based on the PPD. The side on which the test and
reference images are presented shall be randomly assigned on each trial.
Centered and above these images a second copy of the reference image shall be presented. Images may
be arranged in either portrait or landscape orientation.
All images shall be shown with 1:1 mapping to the display pixels without anti-aliasing.
In all trials, a test image is to be compared against its corresponding reference image.
Figure A.1 shows an example display using a cropped portion of Boat as a reference image on top and
as a reference image as one of the choices on the bottom next to a test image. The observer prompt is
below the images.
Figure A.1 — Example display showing image arrangements for the non-flickering test protocol
A.1.2 Image trial ordering
Two consecutive trials shall present a different test image, even if the reference image is the same. For
example, the test image differs in either algorithm or compression level.
During a session, the presentation order of the set of test images shall be randomized.
8 © ISO/IEC 2015 – All rights reserved

A.1.3 Control Image
As part of each block of trials, at least 5 % of the test images should contain defects that a typical
observer would be able to detect.
A.2 Test duration and timing
The time for an observer to complete o
...