ISO 20462-2:2005
(Main)Photography — Psychophysical experimental methods for estimating image quality — Part 2: Triplet comparison method
Photography — Psychophysical experimental methods for estimating image quality — Part 2: Triplet comparison method
ISO 20462-2:2005 defines a standard psychophysical experimental method for subjective image quality assessment of soft copy and hard copy still picture images.
Photographie — Méthodes psychophysiques expérimentales pour estimer la qualité d'image — Partie 2: Méthode comparative du triplet
Fotografija - Psihofizične eksperimentalne metode za ocenjevanje slikovne kakovosti - 2. del: Trojna primerjalna metoda
Ta del ISO 20462 opredeljuje standardno psihofizično eksperimentalno metodo za subjektivno ocenjevanje slikovne kakovosti mirujočih slik v elektronski obliki in na papirju.
General Information
Standards Content (Sample)
SLOVENSKI STANDARD
01-julij-2011
)RWRJUDILMD3VLKRIL]LþQHHNVSHULPHQWDOQHPHWRGH]DRFHQMHYDQMHVOLNRYQH
NDNRYRVWLGHO 7URMQDSULPHUMDOQDPHWRGD
Photography - Psychophysical experimental methods for estimating image quality - Part
2: Triplet comparison method
Photographie - Méthodes psychophysiques expérimentales pour estimer la qualité
d'image - Partie 2: Méthode comparative du triplet
Ta slovenski standard je istoveten z: ISO 20462-2:2005
ICS:
37.040.01 Fotografija na splošno Photography in general
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
INTERNATIONAL ISO
STANDARD 20462-2
First edition
2005-11-01
Photography — Psychophysical
experimental methods for estimating
image quality —
Part 2:
Triplet comparison method
Photographie — Méthodes psychophysiques expérimentales pour
estimer la qualité d'image —
Partie 2: Méthode comparative du triplet
Reference number
©
ISO 2005
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO 2005
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO 2005 – All rights reserved
Contents Page
Foreword. iv
Introduction . v
1 Scope. 1
2 Terms and definitions. 1
3 Two-step psychophysical method. 2
4 Experimental procedure. 3
4.1 Step 1. 3
4.2 Step 2. 3
Annex A (informative) Comparison between a paired comparison and a triplet comparison
technique . 4
Annex B (informative) Number of sample combinations for triplet comparison. 6
Annex C (informative) Standard portrait images . 8
Annex D (informative) Performance of the triplet comparison method. 12
Annex E (informative) Scheffe’s method . 17
Annex F (informative) Conversion of Scheffe’s scale to JND. 22
Bibliography . 25
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies
(ISO member bodies). The work of preparing International Standards is normally carried out through ISO
technical committees. Each member body interested in a subject for which a technical committee has been
established has the right to be represented on that committee. International organizations, governmental and
non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the
International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of technical committees is to prepare International Standards. Draft International Standards
adopted by the technical committees are circulated to the member bodies for voting. Publication as an
International Standard requires approval by at least 75 % of the member bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO shall not be held responsible for identifying any or all such patent rights.
ISO 20462-2 was prepared by Technical Committee ISO/TC 42, Photography.
ISO 20462 consists of the following parts, under the general title Photography — Psychophysical experimental
method for estimating image quality:
⎯ Part 1: Overview of psychophysical elements
⎯ Part 2: Triplet comparison method
⎯ Part 3: Quality ruler method
iv © ISO 2005 – All rights reserved
Introduction
This part of ISO 20462 is necessary to provide a basis for visually assessing photographic image quality in a
precise, repeatable and efficient manner. This part of ISO 20462 is needed in order to evaluate various test
methods or image processing algorithms that may be used in other international and industry standards. For
example, it should be used to perform subjective evaluation of exposure series images from digital cameras
as part of the work needed for future revisions of ISO 12232.
The opportunities to create and observe images using different types of hard copy media and soft copy
displays have increased significantly with advances in computer-based digital imaging technology. As a result,
there is a need to develop requirements for obtaining colour-appearance matches between images produced
using various media and display technologies under a variety of viewing conditions. To develop the necessary
requirements, organizations, including the CIE and the ICC, are developing methods to compensate for the
effect of different viewing conditions, and to map colours optimally across disparate media having different
colour gamuts.
Such technical activities are often faced with the need to evaluate proposed methods or algorithms by visual
[1]
assessment based on psychophysical experiments. K.M. Braun et al. examined five viewing techniques for
cross-media image comparisons in terms of sensitivity of scaling, and mental and physical stress for the
observers. CIE TC1-27 “Specification of Colour Appearance for Reflective Media and Self-Luminous Display
Comparisons” proposed guidelines for conducting psychophysical experiments for the evaluation of
[6]
colorimetric and colour-appearance models . Accordingly, for the design and evaluation of digital imaging
systems, it is of great importance to develop a methodology for subjective visual assessment, so that reliable
and stable results can be derived with minimum observer stress.
When performing a psychophysical experiment, it is highly desirable to obtain results that are precise and
reproducible. In order to derive statistically reliable results, large numbers of observers are required and
careful attention should be paid to the experimental setup. Multiple (repeated) assessments are also useful.
Observer stress during the visual assessment process can adversely affect the results. The order of image
presentation, and the types of questions or questionnaires addressed by the observers, can also affect the
results.
Table 1 gives a comparison of three visual assessment techniques commonly used for image quality
evaluation. The advantages of the category methods include low stress and high stability, since the observer’s
task is to rank each image using typically five or seven categories. However, its scalability within a category is
less precise. One of the most common techniques for image quality assessment is the paired comparison
method. This method is particularly suited to assessing image quality when precise scalability is required.
However, a serious problem with the paired comparison method is that the number of samples to be
examined is to be relatively limited. As the number of the samples increases, the number of combinations
becomes extensive. This causes excessive observer stress, which can affect the accuracy and repeatability of
the results. The third method, commonly known as magnitude scaling, is magnitude estimation. This method
is extremely difficult when the psychophysical experiments are conducted using ordinary (non-expert)
observers to perform the image quality assessment.
Table 1 — Comparison of typical psychophysical experimental methods
Name of method Scalability Stability Stress
Category Low High Low
Magnitude estimation Medium Low Medium
Paired comparison High High High
[3]
G. Johnson et al. have proposed “A sharpness rule”, where the magnitude of sharpness was analyzed in
terms of resolution, contrast, noise and degree of sharpness-enhancement. Likewise, preferred skin colour
may be considered not only from the viewpoint of chromaticity, but also with respect to the lightness,
[4]
background and white point of the display media . These examples show that image quality is not always
evaluated by a single attribute, but may vary in combination with multiple attributes. In cases where a
psychophysical experiment is designed for a new application, the experimenter may need to vary many
attributes simultaneously during the course of the experiment. In these situations, the number of the samples
to be examined becomes excessively large, making it difficult to employ the paired comparison technique.
vi © ISO 2005 – All rights reserved
INTERNATIONAL STANDARD ISO 20462-2:2005(E)
Photography — Psychophysical experimental methods
for estimating image quality —
Part 2:
Triplet comparison method
1 Scope
This part of ISO 20462 defines a standard psychophysical experimental method for subjective image quality
assessment of soft copy and hard copy still picture images.
2 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
2.1
just noticeable difference
JND
stimulus difference that would lead to a 75:25 proportion of responses in a paired comparison task
2.2
psychophysical experimental method
experimental technique for subjective evaluation of image quality or attributes thereof, from which stimulus
differences in units of JNDs may be estimated
cf. categorical sort (2.5), paired comparison (2.3) and triplet comparison methods (2.4)
2.3
paired comparison method
psychophysical method involving the choice of which of two simultaneously presented stimuli exhibits greater
or lesser image quality or an attribute thereof, in accordance with a set of instructions given to the observer
NOTE Two limitations of the paired comparison method are as follows.
a) If all possible stimulus comparisons are done, as is usually the case, a large number of assessments are required for
even modest numbers of experimental stimulus levels [if N levels are to be studied, N(N − 1)/2 paired comparisons
are needed].
b) If a stimulus difference exceeds approximately 1,5 JNDs, the magnitude of the stimulus difference cannot be directly
estimated reliably because the response saturates as the proportions approach unanimity.
However, if a series of stimuli having no large gaps are assessed, the differences between more widely separated stimuli
may be deduced indirectly by summing smaller, reliably determined (unsaturated) stimulus differences. The standard
methods for transformation of paired comparison data to an interval scale (a scale linearly related to JNDs) perform
statistically optimized procedures for inferring the stimulus differences, but they may yield unreliable results when too
many of the stimulus differences are large enough (> 1,5 JNDs) that they produce saturated responses.
2.4
triplet comparison
psychophysical method that involves the simultaneous scaling of three test stimuli with respect to image
quality or an attribute thereof, in accordance with a set of instructions given to the observer
2.5
categorical sort method
psychophysical method involving the classification of a stimulus into one of several ordered categories, at
least some of which are identified by adjectives or phrases that describe different levels of image quality or
attributes thereof
NOTE The application of a
...
INTERNATIONAL ISO
STANDARD 20462-2
First edition
2005-11-01
Photography — Psychophysical
experimental methods for estimating
image quality —
Part 2:
Triplet comparison method
Photographie — Méthodes psychophysiques expérimentales pour
estimer la qualité d'image —
Partie 2: Méthode comparative du triplet
Reference number
©
ISO 2005
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO 2005
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO 2005 – All rights reserved
Contents Page
Foreword. iv
Introduction . v
1 Scope. 1
2 Terms and definitions. 1
3 Two-step psychophysical method. 2
4 Experimental procedure. 3
4.1 Step 1. 3
4.2 Step 2. 3
Annex A (informative) Comparison between a paired comparison and a triplet comparison
technique . 4
Annex B (informative) Number of sample combinations for triplet comparison. 6
Annex C (informative) Standard portrait images . 8
Annex D (informative) Performance of the triplet comparison method. 12
Annex E (informative) Scheffe’s method . 17
Annex F (informative) Conversion of Scheffe’s scale to JND. 22
Bibliography . 25
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies
(ISO member bodies). The work of preparing International Standards is normally carried out through ISO
technical committees. Each member body interested in a subject for which a technical committee has been
established has the right to be represented on that committee. International organizations, governmental and
non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the
International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of technical committees is to prepare International Standards. Draft International Standards
adopted by the technical committees are circulated to the member bodies for voting. Publication as an
International Standard requires approval by at least 75 % of the member bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO shall not be held responsible for identifying any or all such patent rights.
ISO 20462-2 was prepared by Technical Committee ISO/TC 42, Photography.
ISO 20462 consists of the following parts, under the general title Photography — Psychophysical experimental
method for estimating image quality:
⎯ Part 1: Overview of psychophysical elements
⎯ Part 2: Triplet comparison method
⎯ Part 3: Quality ruler method
iv © ISO 2005 – All rights reserved
Introduction
This part of ISO 20462 is necessary to provide a basis for visually assessing photographic image quality in a
precise, repeatable and efficient manner. This part of ISO 20462 is needed in order to evaluate various test
methods or image processing algorithms that may be used in other international and industry standards. For
example, it should be used to perform subjective evaluation of exposure series images from digital cameras
as part of the work needed for future revisions of ISO 12232.
The opportunities to create and observe images using different types of hard copy media and soft copy
displays have increased significantly with advances in computer-based digital imaging technology. As a result,
there is a need to develop requirements for obtaining colour-appearance matches between images produced
using various media and display technologies under a variety of viewing conditions. To develop the necessary
requirements, organizations, including the CIE and the ICC, are developing methods to compensate for the
effect of different viewing conditions, and to map colours optimally across disparate media having different
colour gamuts.
Such technical activities are often faced with the need to evaluate proposed methods or algorithms by visual
[1]
assessment based on psychophysical experiments. K.M. Braun et al. examined five viewing techniques for
cross-media image comparisons in terms of sensitivity of scaling, and mental and physical stress for the
observers. CIE TC1-27 “Specification of Colour Appearance for Reflective Media and Self-Luminous Display
Comparisons” proposed guidelines for conducting psychophysical experiments for the evaluation of
[6]
colorimetric and colour-appearance models . Accordingly, for the design and evaluation of digital imaging
systems, it is of great importance to develop a methodology for subjective visual assessment, so that reliable
and stable results can be derived with minimum observer stress.
When performing a psychophysical experiment, it is highly desirable to obtain results that are precise and
reproducible. In order to derive statistically reliable results, large numbers of observers are required and
careful attention should be paid to the experimental setup. Multiple (repeated) assessments are also useful.
Observer stress during the visual assessment process can adversely affect the results. The order of image
presentation, and the types of questions or questionnaires addressed by the observers, can also affect the
results.
Table 1 gives a comparison of three visual assessment techniques commonly used for image quality
evaluation. The advantages of the category methods include low stress and high stability, since the observer’s
task is to rank each image using typically five or seven categories. However, its scalability within a category is
less precise. One of the most common techniques for image quality assessment is the paired comparison
method. This method is particularly suited to assessing image quality when precise scalability is required.
However, a serious problem with the paired comparison method is that the number of samples to be
examined is to be relatively limited. As the number of the samples increases, the number of combinations
becomes extensive. This causes excessive observer stress, which can affect the accuracy and repeatability of
the results. The third method, commonly known as magnitude scaling, is magnitude estimation. This method
is extremely difficult when the psychophysical experiments are conducted using ordinary (non-expert)
observers to perform the image quality assessment.
Table 1 — Comparison of typical psychophysical experimental methods
Name of method Scalability Stability Stress
Category Low High Low
Magnitude estimation Medium Low Medium
Paired comparison High High High
[3]
G. Johnson et al. have proposed “A sharpness rule”, where the magnitude of sharpness was analyzed in
terms of resolution, contrast, noise and degree of sharpness-enhancement. Likewise, preferred skin colour
may be considered not only from the viewpoint of chromaticity, but also with respect to the lightness,
[4]
background and white point of the display media . These examples show that image quality is not always
evaluated by a single attribute, but may vary in combination with multiple attributes. In cases where a
psychophysical experiment is designed for a new application, the experimenter may need to vary many
attributes simultaneously during the course of the experiment. In these situations, the number of the samples
to be examined becomes excessively large, making it difficult to employ the paired comparison technique.
vi © ISO 2005 – All rights reserved
INTERNATIONAL STANDARD ISO 20462-2:2005(E)
Photography — Psychophysical experimental methods
for estimating image quality —
Part 2:
Triplet comparison method
1 Scope
This part of ISO 20462 defines a standard psychophysical experimental method for subjective image quality
assessment of soft copy and hard copy still picture images.
2 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
2.1
just noticeable difference
JND
stimulus difference that would lead to a 75:25 proportion of responses in a paired comparison task
2.2
psychophysical experimental method
experimental technique for subjective evaluation of image quality or attributes thereof, from which stimulus
differences in units of JNDs may be estimated
cf. categorical sort (2.5), paired comparison (2.3) and triplet comparison methods (2.4)
2.3
paired comparison method
psychophysical method involving the choice of which of two simultaneously presented stimuli exhibits greater
or lesser image quality or an attribute thereof, in accordance with a set of instructions given to the observer
NOTE Two limitations of the paired comparison method are as follows.
a) If all possible stimulus comparisons are done, as is usually the case, a large number of assessments are required for
even modest numbers of experimental stimulus levels [if N levels are to be studied, N(N − 1)/2 paired comparisons
are needed].
b) If a stimulus difference exceeds approximately 1,5 JNDs, the magnitude of the stimulus difference cannot be directly
estimated reliably because the response saturates as the proportions approach unanimity.
However, if a series of stimuli having no large gaps are assessed, the differences between more widely separated stimuli
may be deduced indirectly by summing smaller, reliably determined (unsaturated) stimulus differences. The standard
methods for transformation of paired comparison data to an interval scale (a scale linearly related to JNDs) perform
statistically optimized procedures for inferring the stimulus differences, but they may yield unreliable results when too
many of the stimulus differences are large enough (> 1,5 JNDs) that they produce saturated responses.
2.4
triplet comparison
psychophysical method that involves the simultaneous scaling of three test stimuli with respect to image
quality or an attribute thereof, in accordance with a set of instructions given to the observer
2.5
categorical sort method
psychophysical method involving the classification of a stimulus into one of several ordered categories, at
least some of which are identified by adjectives or phrases that describe different levels of image quality or
attributes thereof
NOTE The application of adjectival descriptors is strongly affected by the range of stimuli presented, so that it is
difficult to compare the results of one categorical sort experiment to another. Range effects and the coarse quantization of
categorical sort experiments also hinder conversion of the responses to JND units. Given these limitations, it is not
possible to unambiguously map adjectival descriptors to JND units, but it is worth noting that in some experiments where a
broad range of stimuli have been presented, the categories excellent, very good, good, fair, poor, and not worth keeping
have been found to provide very roughly comparable intervals that average about six quality JNDs in width.
2.6
observer
individual performing the subjective evaluation task in a psychophysical method
3 Two-step psychophysical method
This part of ISO 20462 defines a new psychophysical experimental method, which satisfies the following
requirements:
⎯ enables a large number of samples to be examined;
⎯ provides pr
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.