ISO/IEC TR 29170-1:2017
(Main)Information technology — Advanced image coding and evaluation — Part 1: Guidelines for image coding system evaluation
Information technology — Advanced image coding and evaluation — Part 1: Guidelines for image coding system evaluation
ISO/IEC TR 29170-1:2017 recommends best practices for coding system evaluation of images and image sequences. ISO/IEC TR 29170-1:2017 defines a common vocabulary of terms for coding system evaluation and divides evaluation methods into three broad categories: a) subjective assessment; b) objective assessment; c) computational assessment. In addition to these broad assessment categories, this document discusses special care that is given for coding unusual imagery, e.g. high dynamic range or high colour depth. A fourth assessment category, hardware complexity, is often important for real-time or computationally complex applications; however, it is outside the scope of this document.
Technologies de l'information — Codage d'image avancé et évaluation — Partie 1: Lignes directices pour l'évaluation des systèmes de codage d'image
General Information
Standards Content (Sample)
TECHNICAL ISO/IEC TR
REPORT 29170-1
First edition
2017-10
Information technology — Advanced
image coding and evaluation —
Part 1:
Guidelines for image coding system
evaluation
Technologies de l'information — Codage d'image avancé et
évaluation —
Partie 1: Lignes directices pour l'évaluation des systèmes de
codage d'image
Reference number
©
ISO/IEC 2017
© ISO/IEC 2017, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2017 – All rights reserved
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Abbreviated terms . 3
5 Selection and characteristics of test images . 4
5.1 Common image characteristics . 4
5.2 Bits per pixel . 5
5.3 Compression ratio . 5
5.4 Variation in bit rates . 5
5.4.1 Constant bit rate systems . 5
5.4.2 Variable bit rate systems . 5
5.5 Error resilience . 6
5.6 Recursive compression assessment . 6
5.7 Image selection . 6
6 Best practices of subjective image quality assessments . 7
6.1 Goals of subjective assessment . 7
6.2 Subjective assessment evaluation procedures . 7
6.2.1 Observer selection . 7
6.2.2 Visual acuity . 7
6.2.3 Number of observers . 7
6.2.4 Instructions to observers . 8
6.2.5 Evaluation scales . 8
6.2.6 Statistical analysis . . 8
6.3 Viewing conditions for electronic displays . 8
6.3.1 Purpose . 8
6.3.2 ISO 3664 . 9
6.3.3 ISO 9241 . 9
6.4 Goals for evaluation of visually lossless and nearly lossless coding . 9
7 Best practices of objective image quality assessment methodology .9
Annex A (informative) Subjective metrics .11
Annex B (informative) Objective metrics .14
Annex C (informative) Computational metrics .19
Annex D (informative) Verification of codec characteristics .31
Bibliography .34
© ISO/IEC 2017 – All rights reserved iii
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the following
URL: www.iso.org/iso/foreword.html.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 29170 series can be found on the ISO website.
iv © ISO/IEC 2017 – All rights reserved
Introduction
This document provides a framework and best practices to evaluate image compression algorithms.
This document provides a selection of evaluation tools that allow testing multiple features, including
objective metric image quality, subjective metric image quality and codec algorithmic complexity.
Which features of codecs should be tested and pass-fail criteria is beyond the scope of this document.
© ISO/IEC 2017 – All rights reserved v
TECHNICAL REPORT ISO/IEC TR 29170-1:2017(E)
Information technology — Advanced image coding and
evaluation —
Part 1:
Guidelines for image coding system evaluation
1 Scope
This document recommends best practices for coding system evaluation of images and image
sequences. This document defines a common vocabulary of terms for coding system evaluation and
divides evaluation methods into three broad categories:
a) subjective assessment;
b) objective assessment;
c) computational assessment.
In addition to these broad assessment categories, this document discusses special care that is given for
coding unusual imagery, e.g. high dynamic range or high colour depth.
A fourth assessment category, hardware complexity, is often important for real-time or computationally
complex applications; however, it is outside the scope of this document.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— IEC Electropedia: available at http://www.electropedia.org/
— ISO Online browsing platform: available at http://www.iso.org/obp
3.1
channel
one logical component of an image
Note 1 to entry: A channel may be a direct representation of one component from the bitstream, or may be
generated by the application of a palette to a component from the bitstream.
[SOURCE: ISO/IEC 15444-1:2016, 3.17 – modified to move part of definition into a Note to entry]
3.2
codec
coding system
system comprising a compressor (3.6), a decompressor (3.8) and the compressor's bitstream output is
compatible with the decompressor's bitstream input
© ISO/IEC 2017 – All rights reserved 1
3.3
component
two-dimensional array of samples
Note 1 to entry: An image typically consists of several components, for instance, representing red, green, and blue.
[SOURCE: ISO/IEC 15444-1:2016, 3.26 – modified to move part of definition into a Note to entry]
3.4
component bit depth
number of bits of precision of colour channels (or components) of an unencoded image
3.5
component number
number of colour channels (or components) encoded in an image
3.6
compressor
portion of a coding system that has a pixel stream and may have control metadata as its input and a
coded bitstream as its output
3.7
constant bit rate
mode where the number of encoded bits from a portion of an image represented by a fixed number of
pixels (3.16) does not vary compared to the number of encoded bits in any other equally sized portion
of the same image
3.8
decompressor
portion of a codec (coding system) (3.2) that has a coded bitstream as its input and a pixel (3.16) stream
as its output
3.9
drift
net generational loss of image quality if the output of a lossy image compression/reconstruction cycle is
recompressed again under the same conditions by the same codec (3.2)
3.10
expert observer
observer that has expertise in image artefacts that may be introduced by the system under test or who
has designed or participated in the selection of test content for the system under test
3.11
generational quality loss
measure of quality loss (3.17) between a reference image and a reconstruction of the same image after
repetitive generations of encoding and decoding
3.12
horizontal pixel resolution
horizontal extent of the image in image pixels (3.16) where the horizontal extent may depend on the
channel
3.13
idempotent
codec (3.2) that operates losslessly on its own decompression output
3.14
non-expert observer
naïve observer
observer that has no expertise in the image artefacts that may be introduced by the system under test
2 © ISO/IEC 2017 – All rights reserved
ISO/IEC TR 29
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.