ISO/TR 19247:2016

Photography - Guidelines for reliable testing of digital still cameras

ISO/TR 19247:2016 is applied to the reliable testing of digital cameras (DSCs).

Photographie — Lignes directrices pour des essais fiables des caméras numériques

General Information

• Status: Published
• Publication Date: 09-Jun-2016
• ICS: 37.040.10 - Photographic equipment. Projectors
• Technical Committee: ISO/TC 42 - Photography
• Drafting Committee: ISO/TC 42/WG 18 - Electronic still picture imaging
• Current Stage: 6060 - International Standard published
• Start Date: 10-Jun-2016
• Due Date: 12-Nov-2017
• Completion Date: 12-Nov-2017

Overview

ISO/TR 19247:2016 - Photography: Guidelines for reliable testing of digital still cameras (DSCs) provides practical guidance for conducting credible, repeatable tests of digital cameras. It addresses both subjective (visual) evaluation and objective technical measurements, and is written to help testers, reviewers and manufacturers - including those without a photographic or deep scientific background - produce and interpret reliable camera test results.

Key Topics and Requirements

• Scope: Guidance for reliable testing of digital still cameras (DSC testing), covering procedures, documentation and interpretation.
• Visual analysis (subjective testing):
  • Use calibrated and profiled display/printer devices with a consistent colour-management workflow.
  • Ensure test persons meet basic visual criteria (e.g., screening for colour deficiency) and represent a mix of ages/sexes.
  • Minimum of 5 test persons recommended; fewer than 5 is generally insufficient. If under 20 participants, have each person evaluate images twice.
  • Use reference “good” and “bad” images, randomize image order, and limit test cycles to about 15–25 images with breaks between cycles.
  • Select test motifs that highlight the attribute being evaluated (colour, dynamic range, sharpness, chromatic aberration, distortion, flare, face recognition, etc.).
• Technical measurements (objective testing):
  • A comprehensive catalogue of camera performance aspects is provided - testers should choose a subset relevant to the camera’s intended use.
  • Suggested measurable criteria include resolution (ISO 12233), texture/low-contrast detail (ISO 19567-1), opto-electronic conversion (OECF, ISO 14524), dynamic range, noise (ISO 15739), colour accuracy (ISO 17321 / CIE), geometric distortion (ISO 17850), vignetting (ISO 17957), chromatic aberration (ISO 19084), shutter lag (ISO 15781), compression artefacts, low-light performance, stray light/flare, defect pixels and battery life (ISO 20087).
• Test conditions and documentation:
  • Choose camera settings and environmental conditions that reflect common photographic use.
  • Archive images, test charts and software; disclose measurement methods and mathematical analysis for transparency.
  • Minimize deviations through consistent procedures and trained personnel.

Applications and Who Uses It

• Camera manufacturers for validation and product development.
• Independent test labs and QA teams for repeatable performance benchmarking.
• Photography magazines, reviewers and test-equipment vendors to improve credibility of published comparisons.
• Software/tool developers creating evaluation or analysis tools for DSCs.
• Professional photographers and procurement teams assessing camera suitability for real-world tasks.

Related Standards

ISO/TR 19247:2016 references and aligns with several measurement standards, including:

• ISO 12233 (resolution/MTF)
• ISO 14524 (OECF)
• ISO 15739 (noise, dynamic range)
• ISO 17321 / CIE standards (colour reproduction)
• ISO 17850 (distortion), ISO 17957 (vignetting), ISO 19084 (chromatic displacement)
• ISO 20087 (battery life)

Using ISO/TR 19247:2016 helps ensure transparent, repeatable and credible digital still camera testing, improving comparability and trust in image quality and camera performance reports.

Standards Content (Sample)

TECHNICAL ISO/TR
REPORT 19247
First edition
Photography — Guidelines for reliable
testing of digital still cameras
Photographie — Guide pour le fiable essais des caméras numériques
PROOF/ÉPREUVE
Reference number
©
ISO 2016
© ISO 2016, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. [phone]
Fax [phone]
[e-mail]
www.iso.org
ii © ISO 2016 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Tests based on visual analysis . 1
2.1 General . 1
2.2 Tests under conditions which are not replicable . 2
2.3 Simple tests under conditions which are replicable (with few test persons) . 2
3 Tests based on technical measurements . 3
3.1 Catalogue of individual technical aspects and suggestions . 3
3.1.1 General. 3
3.2 Test criteria . 4
3.2.1 General. 4
3.3 Test conditions . 4
3.4 Software and samples . 5
3.4.1 Test charts . 5
3.4.2 Evaluation of images . . 5
3.5 Archiving and documentation . 6
3.6 Minimization of deviations . 6
3.7 Preparation of personnel for reliable testing . 6
Bibliography . 8
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity assessment,
as well as information about ISO’s adherence to the World Trade Organization (WTO) principles in the
Technical Barriers to Trade (TBT) see the following URL: www.iso.org/iso/foreword.html.
The committee responsible for this document is ISO/TC 42, Photography.
iv PROOF/ÉPREUVE © ISO 2016 – All rights reserved

Introduction
The motivation for producing these guidelines for testing digital still cameras (DSCs) is the increasing
use of tests by industries and the press. On one hand, the booming DSC market attracts published
evaluations and comparisons of its products. On the other hand, complex and, more importantly, reliable
testing needs expensive test equipment and experienced and knowledgeable testing technicians. This
guide provides direction and procedures for reliable testing of DSCs for those people with neither a
photographic nor a scientific background.
The growing commercial development of DSC test methods and equipment is additional motivation for
producing these guidelines as it is often difficult to make specific assessments about the usability of
new test equipment and methods and about the reliability of their results. For the credibility of test
methods and equipment the disclosure of the measurement methods and the mathematical analysis
used by the manufacturer of the test equipment is important.
These suggestions follow two goals in order to improve the current situation for the parties involved:
a) Technical guidelines for the testing process
This guide gives an overview of the important, useful and useable test criteria, procedures and
basic evaluation for the users and producers of test methods. It will help to avoid typical mistakes
and will improve and secure the validity and reliability of the used methods.
b) Guidelines for the interpretation of test results
At the moment, there are no guidelines by which the receiver and user of test results, especially
magazine readers and digital camera producers, can estimate the reliability and practicability, or
in extreme cases the credibility, of test results. This guide will provide support in securing a basis
for the assessment, analysis and even discussion with the testers. Therefore, this guide will provide
more transparency in this complex area.
TECHNICAL REPORT ISO/TR 19247:2016(E)
Photography — Guidelines for reliable testing of digital
still cameras
1 Scope
This Technical Report is applied to the reliable testing of digital cameras (DSCs).
2 Tests based on visual analysis
2.1 General
The visual analysis of digital image data using test persons is not easy as it seems. This becomes more
evident when one considers that digital images are not visible. In order to visualize an image, a device
needs to be used, i.e. a monitor or a printer.
In order for the correct interpretation of the image data to be possible, it is important to assure that
the visible representation created by the device doesn’t have any imponderable or even unknown
limitations. This means that the devices needs to be calibrated and profiled, and with the help of a
suitable colour management workflow allow the correct reproduction of the image. The tester needs
to know about the limitations that can be introduced by the device especially in the areas of colour
and resolution. Use DIN 15708-1 as a guideline to find out if an acceptably good colour reproduction is
available.
It is important to consider that in some cases not all colours from the test specimen can be correctly
captured and that colours, which are accurately captured, are not always correctly reproduced on the
display or paper.
As a prerequisite for the test person, their ability to see especially with reference to dyschromatopsia
(see Ishihara Test) and lack of 20/20 vision shall be given and tested. It is preferable that the test persons
be composed of people of different sex and age, so that a representative selection of basic population
can be achieved. When the testing starts, all test persons need to be relaxed and healthy. ISO 20462-3
describes the recommended procedure for subjective evaluation in the “quality ruler method”.
Table 1 shows the magnitude of the influence of a single person in a group of persons on the median of
the subjectively acquired evaluation results. This is derived from the standard deviation:
n n
2 2
1 1
s = xx− = xx− (1)
() ()
∑∑
i i
n−1 n−1
i==1 i 1
The factor e = in Table 1 represents the impact of a single person on the result based on the
n− 1
total number of observers. It is the standard deviation normalized to the square root of the sum of all
squared deviations.
s 1
e = = (2)
n−1
n 2
xx−
()
∑
i
i=1
Table 1 — Influence of a single measurement on median of subjective results
Number of test persons e
1 ∞
2 1,00
3 0,71
4 0,58
5 0,50
10 0,33
20 0,23
30 0,19
50 0,14
100 0,10
The number of test persons needs to be large enough. Using less than 5 subjects is generally insufficient
for a qualified evaluation.
2.2 Tests under conditions which are not replicable
It is preferable that tests fulfil as many of the criteria outlined in 3.2 as possible so that the results
are as meaningful and useful as possible. When using a comparative product test, those tests made
under conditions that are not replicable are not acceptable. Tests run under conditions that are not
replicable are usable only be used as a rough indication of the image quality and are not suitable for
reliable conclusions.
2.3 Simple tests under conditions which are replicable (with few test persons)
It is preferable that the following conditions are ensured by those responsible for the testing of the
evaluation of image quality under conditions that are replicable.
a) The image display equipment needs to be calibrated and profiled accordingly.
b) Provide complete and accurate documentation for the colour space, reproduction devices, and
reproduction media used.
c) Test persons are at least 5.
d) The image quality evaluation scale needs to be clearly defined before beginning the tests.
e) All test criteria are to be collected on the same evaluation scale in order to avoid mistakes and
confusing the test persons.
f) As part of the test and before evaluators are asked to evaluate image samples, it is suggested that
the evaluators are shown reference samples of nominally excellent quality and nominally poor
quality on the quality evaluation scale with an explanation of images captured by the DSC are
ranked with respect to these reference images.
g) The image quality aspect that is to be evaluated, needs to be represented in the motif captured
by the device under test. Scenes that illustrate one or more of the following image quality aspects
should be identified or created. Sample images for reference and evaluation should be made from
these scenes.
1) For the subjective evaluation of colour, an adequate number of relevant colours needs to be
present in the photographed objects. This especially includes the green tones of different
plants, different light and darker skin tones and blue tones, which occur in pictures of the sky
when the sun is shining; as well as grey tones and highly saturated colours, for example in
flowers. In addition, colour, which are mixed from the primary colours of cyan, magenta or
2 PROOF/ÉPREUVE © ISO 2016 – All rights reserved

yellow represented in different brightness and saturation levels are useful because these often
show significant deviations between the original and the captured image.
2) For the subjective evaluation of the dynamic range in the images, specular highlights reflecting
off metallic surfaces and dark structures in shaded image areas can be useful. Sufficient detail
is also important in the mid tones of the image.
3) For the subjective evaluation of the image sharpness, fine structures distributed throughout
the image, including the corners of the image are useful, especially structures with varying
orientation (for example Siemens stars or other contrast rich patterns like, for example, the
bristles on a brush).
4) If the corner shading is to be subjectively evaluated, an evenly lit partial area in the image that
stretches from the centre to the image corners is important.
5) The evaluation of distortion in the image needs a regular grid of structures such as straight
lines, which are suggested to be horizontally and vertically oriented (for example, in quadrangle
test charts or edges of buildings).
6) If the chr
...

TECHNICAL ISO/TR
REPORT 19247
First edition
2016-07-01
Photography — Guidelines for reliable
testing of digital still cameras
Photographie — Lignes directrices pour des essais fiables des caméras
numériques
Reference number
©
ISO 2016
© ISO 2016, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. [phone]
Fax [phone]
[e-mail]
www.iso.org
ii © ISO 2016 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Tests based on visual analysis . 1
2.1 General . 1
2.2 Tests under conditions which are not replicable . 2
2.3 Simple tests under conditions which are replicable (with few test persons) . 2
3 Tests based on technical measurements . 3
3.1 Catalogue of individual technical aspects and suggestions . 3
3.1.1 General. 3
3.2 Test criteria . 4
3.2.1 General. 4
3.3 Test conditions . 4
3.4 Software and samples . 5
3.4.1 Test charts . 5
3.4.2 Evaluation of images . . 5
3.5 Archiving and documentation . 6
3.6 Minimization of deviations . 6
3.7 Preparation of personnel for reliable testing . 6
Bibliography . 8
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity assessment,
as well as information about ISO’s adherence to the World Trade Organization (WTO) principles in the
Technical Barriers to Trade (TBT) see the following URL: www.iso.org/iso/foreword.html.
The committee responsible for this document is ISO/TC 42, Photography.
iv © ISO 2016 – All rights reserved

Introduction
The motivation for producing these guidelines for testing digital still cameras (DSCs) is the increasing
use of tests by industries and the press. On one hand, the booming DSC market attracts published
evaluations and comparisons of its products. On the other hand, complex and, more importantly, reliable
testing needs expensive test equipment and experienced and knowledgeable testing technicians. This
guide provides direction and procedures for reliable testing of DSCs for those people with neither a
photographic nor a scientific background.
The growing commercial development of DSC test methods and equipment is additional motivation for
producing these guidelines as it is often difficult to make specific assessments about the usability of
new test equipment and methods and about the reliability of their results. For the credibility of test
methods and equipment the disclosure of the measurement methods and the mathematical analysis
used by the manufacturer of the test equipment is important.
These suggestions follow two goals in order to improve the current situation for the parties involved:
a) Technical guidelines for the testing process
This guide gives an overview of the important, useful and useable test criteria, procedures and
basic evaluation for the users and producers of test methods. It will help to avoid typical mistakes
and will improve and secure the validity and reliability of the used methods.
b) Guidelines for the interpretation of test results
At the moment, there are no guidelines by which the receiver and user of test results, especially
magazine readers and digital camera producers, can estimate the reliability and practicability, or
in extreme cases the credibility, of test results. This guide will provide support in securing a basis
for the assessment, analysis and even discussion with the testers. Therefore, this guide will provide
more transparency in this complex area.
TECHNICAL REPORT ISO/TR 19247:2016(E)
Photography — Guidelines for reliable testing of digital
still cameras
1 Scope
This Technical Report is applied to the reliable testing of digital cameras (DSCs).
2 Tests based on visual analysis
2.1 General
The visual analysis of digital image data using test persons is not easy as it seems. This becomes more
evident when one considers that digital images are not visible. In order to visualize an image, a device
needs to be used, i.e. a monitor or a printer.
In order for the correct interpretation of the image data to be possible, it is important to assure that
the visible representation created by the device doesn’t have any imponderable or even unknown
limitations. This means that the devices needs to be calibrated and profiled, and with the help of a
suitable colour management workflow allow the correct reproduction of the image. The tester needs
to know about the limitations that can be introduced by the device especially in the areas of colour
and resolution. Use DIN 15708-1 as a guideline to find out if an acceptably good colour reproduction is
available.
It is important to consider that in some cases not all colours from the test specimen can be correctly
captured and that colours, which are accurately captured, are not always correctly reproduced on the
display or paper.
As a prerequisite for the test person, their ability to see especially with reference to dyschromatopsia
(see Ishihara Test) and lack of 20/20 vision shall be given and tested. It is preferable that the test persons
be composed of people of different sex and age, so that a representative selection of basic population
can be achieved. When the testing starts, all test persons need to be relaxed and healthy. ISO 20462-3
describes the recommended procedure for subjective evaluation in the “quality ruler method”.
Table 1 shows the magnitude of the influence of a single person in a group of persons on the median of
the subjectively acquired evaluation results. This is derived from the standard deviation:
n n
2 2
1 1
s = xx− = xx− (1)
() ()
∑∑
i i
n−1 n−1
i==1 i 1
The factor e = in Table 1 represents the impact of a single person on the result based on the
n− 1
total number of observers. It is the standard deviation normalized to the square root of the sum of all
squared deviations.
s 1
e = = (2)
n−1
n 2
xx−
()
∑
i
i=1
Table 1 — Influence of a single measurement on median of subjective results
Number of test persons e
1 ∞
2 1,00
3 0,71
4 0,58
5 0,50
10 0,33
20 0,23
30 0,19
50 0,14
100 0,10
The number of test persons needs to be large enough. Using less than 5 subjects is generally insufficient
for a qualified evaluation.
2.2 Tests under conditions which are not replicable
It is preferable that tests fulfil as many of the criteria outlined in 3.2 as possible so that the results
are as meaningful and useful as possible. When using a comparative product test, those tests made
under conditions that are not replicable are not acceptable. Tests run under conditions that are not
replicable are usable only be used as a rough indication of the image quality and are not suitable for
reliable conclusions.
2.3 Simple tests under conditions which are replicable (with few test persons)
It is preferable that the following conditions are ensured by those responsible for the testing of the
evaluation of image quality under conditions that are replicable.
a) The image display equipment needs to be calibrated and profiled accordingly.
b) Provide complete and accurate documentation for the colour space, reproduction devices, and
reproduction media used.
c) Test persons are at least 5.
d) The image quality evaluation scale needs to be clearly defined before beginning the tests.
e) All test criteria are to be collected on the same evaluation scale in order to avoid mistakes and
confusing the test persons.
f) As part of the test and before evaluators are asked to evaluate image samples, it is suggested that
the evaluators are shown reference samples of nominally excellent quality and nominally poor
quality on the quality evaluation scale with an explanation of images captured by the DSC are
ranked with respect to these reference images.
g) The image quality aspect that is to be evaluated, needs to be represented in the motif captured
by the device under test. Scenes that illustrate one or more of the following image quality aspects
should be identified or created. Sample images for reference and evaluation should be made from
these scenes.
1) For the subjective evaluation of colour, an adequate number of relevant colours needs to be
present in the photographed objects. This especially includes the green tones of different
plants, different light and darker skin tones and blue tones, which occur in pictures of the sky
when the sun is shining; as well as grey tones and highly saturated colours, for example in
flowers. In addition, colour, which are mixed from the primary colours of cyan, magenta or
2 © ISO 2016 – All rights reserved

yellow represented in different brightness and saturation levels are useful because these often
show significant deviations between the original and the captured image.
2) For the subjective evaluation of the dynamic range in the images, specular highlights reflecting
off metallic surfaces and dark structures in shaded image areas can be useful. Sufficient detail
is also important in the mid tones of the image.
3) For the subjective evaluation of the image sharpness, fine structures distributed throughout
the image, including the corners of the image are useful, especially structures with varying
orientation (for example Siemens stars or other contrast rich patterns like, for example, the
bristles on a brush).
4) If the corner shading is to be subjectively evaluated, an evenly lit partial area in the image that
stretches from the centre to the image corners is important.
5) The evaluation of distortion in the image needs a regular grid of structures such as straight
lines, which are suggested to be horizontally and vertically oriented (for example, in quadrangle
test charts or edges o
...