ASTM E1499-16(2023)

Standard Guide for Selection, Evaluation, and Training of Observers

SIGNIFICANCE AND USE
5.1 The term appearance (see 3.2.1) implies the essential presence of human visual observations. The results of visual observation involve not only the step of observing, accomplished by the eye, but also the inseparable step of interpretation in the brain. Instrumental test methods currently cannot duplicate this second step, and therefore can now only approximate, but not fully measure, appearance. Such instrumental measures of appearance properties are useful only to the extent that they can be correlated to the results of visual observations by observers of the appearance phenomena being evaluated.  
5.2 Almost invariably, too little attention has been paid to ensuring that the essential visual observations have been properly obtained to provide the basis for correlating visual and instrumental test results.  
5.3 This guide provides the means for assessing observers, by outlining the requirements and tests for their selection, evaluation, and training. This guide should be useful to all experimenters designing or using visual test methods to provide either direct results in terms of the observation of appearance properties, or the experiments correlating such results with instrumental measures approximating the same appearance properties. The user is cautioned to avoid the substitution of validated vision tests with replicas of any kind, either printed, photographed or digitally displayed.
SCOPE
1.1 This guide describes criteria and tests for selecting, evaluating, and training human visual-sensory observers for tasks involving the perception and scaling of properties and phenomena relating to appearance.  
1.2 Examples of tests requiring the use of trained observers include but are not limited to those described in the following ASTM standards: on color, Practice D1535 and Practice E1360; on color difference, Practice D1729 and Test Method D2616; on gloss, Test Method D4449; on metamerism, Practice D4086; and on setting tolerances, Practice D3134.  
1.3 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.4 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

General Information

• Status: Published
• Publication Date: 31-May-2023
• ICS: 13.100 - Occupational safety. Industrial hygiene
• Technical Committee: E12 - Color and Appearance
• Drafting Committee: E12.11 - Visual Methods

Relations

• Refers: ASTM D4086-18(2023) - Standard Practice for Visual Evaluation of Metamerism
• Effective Date: 01-Nov-2023
• Refers: ASTM E1360-05(2019) - Standard Practice for Specifying Color by Using the Optical Society of America Uniform Color Scales System
• Effective Date: 01-Nov-2019
• Refers: ASTM D2616-19 - Standard Test Method for Evaluation of Visual Color Difference With a Gray Scale
• Effective Date: 01-Nov-2019
• Refers: ASTM D3134-15(2019) - Standard Practice for Establishing Color and Gloss Tolerances
• Effective Date: 01-Nov-2019
• Refers: ASTM D4086-18 - Standard Practice for Visual Evaluation of Metamerism
• Effective Date: 01-Oct-2018
• Refers: ASTM E1360-05(2015) - Standard Practice for Specifying Color by Using the Optical Society of America Uniform Color Scales System
• Effective Date: 01-May-2015
• Refers: ASTM D3134-15 - Standard Practice for Establishing Color and Gloss Tolerances
• Effective Date: 01-Jan-2015
• Refers: ASTM D1535-14 - Standard Practice for Specifying Color by the Munsell System
• Effective Date: 01-Nov-2014
• Refers: ASTM E284-13b - Standard Terminology of Appearance
• Effective Date: 01-Nov-2013
• Refers: ASTM E284-13a - Standard Terminology of Appearance
• Effective Date: 01-Jun-2013
• Refers: ASTM E284-13 - Standard Terminology of Appearance
• Effective Date: 01-Jan-2013
• Refers: ASTM D1535-13 - Standard Practice for Specifying Color by the Munsell System
• Effective Date: 01-Jan-2013
• Refers: ASTM D1535-12a - Standard Practice for Specifying Color by the Munsell System
• Effective Date: 15-Jul-2012
• Refers: ASTM D4086-92a(2012) - Standard Practice for Visual Evaluation of Metamerism
• Effective Date: 01-Jul-2012
• Refers: ASTM D2616-12 - Standard Test Method for Evaluation of Visual Color Difference With a Gray Scale
• Effective Date: 01-Jul-2012

Overview

ASTM E1499-16(2023): Standard Guide for Selection, Evaluation, and Training of Observers is a key standard developed by ASTM International. This guide establishes criteria and methods for selecting, assessing, and training human visual-sensory observers for tasks related to the perception and evaluation of appearance attributes, such as color, gloss, metamerism, and visual color difference.

Visual observation remains critical in evaluating appearance properties since current instrumental test methods cannot fully replicate the human interpretation of visual stimuli. The standard aims to improve consistency, accuracy, and validity when using human observers for visual tests or correlating their results with instrumental measurements. This is essential for reliable outcomes in industries where sensory evaluation of appearance plays a pivotal role.

Key Topics

• Selection of Observers: Outlines criteria and visual tests (e.g., color vision and discrimination tests) necessary for identifying qualified observers. This includes using pseudoisochromatic plates, the Farnsworth-Munsell 100 Hue Test, triangle tests, and magnitude scaling tasks to rule out deficiencies and assess the ability to perceive and scale subtle appearance differences.
• Evaluation Procedures: Emphasizes the need for systematic evaluation of observers’ abilities, comparing their performance across specific tasks and tests to ensure suitability for appearance evaluations.
• Training Requirements: Describes conducting training sessions under actual experiment conditions to familiarize observers with specific requirements and tasks, thereby reducing variability and learning effects.
• Precautions in Use: Stresses the importance of controlling experimental variables (lighting, specimen size, test duration) and monitoring observer fatigue and adaptation to maintain the reliability and accuracy of visual assessments.
• Test Design Considerations: Offers recommendations for the number of observers, controlling bias and precision, and ensuring consistent results across observations.

Applications

ASTM E1499-16(2023) is valuable for any application that requires rigorous human visual assessment of appearance properties. Key use areas include:

• Color Matching and Evaluation: In industries such as paint, textiles, and printing, where precise visual color matching is crucial, this guide helps ensure observer competence.
• Gloss and Surface Appearance: Used in quality control of coatings, plastics, and finishes where gloss or surface uniformity is visually assessed.
• Product Development and Testing: Essential for sensory panels and R&D teams performing visual comparison and scaling of appearance characteristics.
• Combination with Instrumental Methods: Supports correlating human visual results with instrumental measures, providing a necessary reference for calibrating or validating instruments.

By following the guidelines outlined in ASTM E1499, organizations can achieve higher reliability and validity in visual assessments, reduce variability in subjective judgments, and enhance inter-laboratory consistency.

Related Standards

ASTM E1499-16(2023) references and complements several related standards focused on visual assessment and appearance testing, including:

• ASTM D1535: Practice for Specifying Color by the Munsell System
• ASTM E1360: Practice for Specifying Color by Using the Optical Society of America Uniform Color Scales System
• ASTM D1729: Practice for Visual Appraisal of Colors and Color Differences of Diffusely-Illuminated Opaque Materials
• ASTM D2616: Test Method for Evaluation of Visual Color Difference With a Gray Scale
• ASTM D4449: Test Method for Visual Evaluation of Gloss Differences Between Surfaces of Similar Appearance
• ASTM D4086: Practice for Visual Evaluation of Metamerism
• ASTM D3134: Practice for Establishing Color and Gloss Tolerances

These standards, used together, establish a robust framework for visual evaluation, ensuring best practices in the selection and training of observers, and the assessment of appearance attributes in industrial and laboratory settings.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E1499 − 16 (Reapproved 2023)
Standard Guide for
Selection, Evaluation, and Training of Observers
This standard is issued under the fixed designation E1499; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope ences Between Surfaces of Similar Appearance
E284 Terminology of Appearance
1.1 This guide describes criteria and tests for selecting,
E1360 Practice for Specifying Color by Using the Optical
evaluating, and training human visual-sensory observers for
Society of America Uniform Color Scales System
tasks involving the perception and scaling of properties and
phenomena relating to appearance.
3. Terminology
1.2 Examples of tests requiring the use of trained observers
3.1 Definitions—Definitions of appearance terms in Termi-
include but are not limited to those described in the following
nology E284 are applicable to this guide.
ASTM standards: on color, Practice D1535 and Practice
3.2 Definitions of Terms Specific to This Standard:
E1360; on color difference, Practice D1729 and Test Method
3.2.1 appearance, n—in psychophysical studies, perception
D2616; on gloss, Test Method D4449; on metamerism, Prac-
in which the spectral and geometric aspects of a visual stimulus
tice D4086; and on setting tolerances, Practice D3134.
are integrated with its illuminating and viewing environment.
1.3 This standard does not purport to address all of the
3.2.2 observer, n—one who judges visually, qualitatively or
safety concerns, if any, associated with its use. It is the
quantitatively, the content of one or more appearance attributes
responsibility of the user of this standard to establish appro-
in each member of a set of objects or stimuli.
priate safety, health, and environmental practices and deter-
3.2.3 scale, v—to assess the content of one or more appear-
mine the applicability of regulatory limitations prior to use.
ance attributes in the members of a set of stimuli.
1.4 This international standard was developed in accor-
3.2.3.1 Discussion—Alternatively, scales may be deter-
dance with internationally recognized principles on standard-
mined by assessing the difference in content of an attribute
ization established in the Decision on Principles for the
with respect to the differences in that attribute among the
Development of International Standards, Guides and Recom-
members of the set.
mendations issued by the World Trade Organization Technical
Barriers to Trade (TBT) Committee.
4. Summary of Guide
4.1 This guide provides descriptions of techniques and tests
2. Referenced Documents
for the selection of candidates for observers for use in visual
2.1 ASTM Standards:
testing, for the evaluation of their capabilities in this field, and
D1535 Practice for Specifying Color by the Munsell System
for their training to enhance these capabilities.
D1729 Practice for Visual Appraisal of Colors and Color
4.2 Discussion is provided of precautions required for the
Differences of Diffusely-Illuminated Opaque Materials
efficient use of observers in visual tests, including avoidance of
D2616 Test Method for Evaluation of Visual Color Differ-
overtaxing the observers and the control of test variables.
ence With a Gray Scale
D3134 Practice for Establishing Color and Gloss Tolerances
4.3 Other considerations of test design, including the num-
D4086 Practice for Visual Evaluation of Metamerism
bers of observers and observations required and the precision
D4449 Test Method for Visual Evaluation of Gloss Differ-
of the visual results, are to be covered elsewhere.
5. Significance and Use
This guide is under the jurisdiction of ASTM Committee E12 on Color and
5.1 The term appearance (see 3.2.1) implies the essential
Appearance and is the direct responsibility of Subcommittee E12.11 on Visual
presence of human visual observations. The results of visual
Methods.
observation involve not only the step of observing, accom-
Current edition approved June 1, 2023. Published July 2023. Originally approved
plished by the eye, but also the inseparable step of interpreta-
in 1992. Last previous edition approved in 2016 as E1499 – 16. DOI: 10.1520/
E1499-23.
tion in the brain. Instrumental test methods currently cannot
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
duplicate this second step, and therefore can now only
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM
approximate, but not fully measure, appearance. Such instru-
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website. mental measures of appearance properties are useful only to the
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E1499 − 16 (2023)
extent that they can be correlated to the results of visual value and chroma, and the observer’s failure is usually by wide
observations by observers of the appearance phenomena being margins in one or more limited regions of the hue circle. The
evaluated. presence of such abnormal results of the test should be grounds
for dismissing the candidate observer.
5.2 Almost invariably, too little attention has been paid to
ensuring that the essential visual observations have been
6.3 Visual Acuity and Discrimination Tests—Having deter-
properly obtained to provide the basis for correlating visual and
mined that the candidate observers have normal color vision, it
instrumental test results.
is next necessary to test their level of discrimination of small
differences in color or another appearance attribute of interest.
5.3 This guide provides the means for assessing observers,
by outlining the requirements and tests for their selection,
6.3.1 Farnsworth-Munsell 100 Hue Test—Use of the
evaluation, and training. This guide should be useful to all Farnsworth-Munsell 100 Hue Test as a color-discrimination
experimenters designing or using visual test methods to pro-
test does not require readministration of the test, but merely
vide either direct results in terms of the observation of
reexamination of the test results. For the purposes of assessing
appearance properties, or the experiments correlating such
color (more precisely, hue) discrimination, the test results are
results with instrumental measures approximating the same
examined for the presence of an approximately constant but
appearance properties. The user is cautioned to avoid the
significant error level in the arrangement of the test chips
substitution of validated vision tests with replicas of any kind,
throughout the hue circle. This may be interpreted as an
either printed, photographed or digitally displayed.
inability to discriminate the small color differences between
neighboring chips. While a weakness of this type might, for
6. Selection and Evaluation of Observers
example, interfere with an observer’s ability to participate in
threshold scaling experiments, the observer might still be
6.1 The process used for selecting observers depends a great
competent to perform magnitude scaling of larger differences
deal upon the type of experiment being carried out, but should
among specimens.
essentially evaluate the potential capability of the observer to
execute a series of visual evaluation tasks (1, 2). When these
6.3.2 Triangle Test—This test is part of a series known as
tasks involve appearance attributes, color or related spectral the Japanese Color Aptitude Test. The candidate observers are
phenomena are often among the task subjects, and if instead
shown, one at a time, a series of 20 sets of three colored chips
geometric phenomena such as gloss are involved, many of the
each. In each set, two of the chips are identical and the third is
same considerations apply. Accordingly, the emphasis in this
slightly different in color. The observer is asked to identify
guide is upon selecting observers for color-related measure-
which one is different, the differences being so small that there
ments. Thus, observers must be screened to rule out those with
is considerable uncertainty in the judgment. A lower than
any color- or task-oriented deficiencies.
average score in this test indicates that the observer does not
differentiate small differences well.
6.2 Color Vision Tests (3):
6.2.1 Pseudoisochromatic Plates—As a preliminary color
6.4 Magnitude Scaling Tests:
vision test, a pseudoisochromatic-plate test should be admin-
6.4.1 Length Estimation—A simple magnitude-scaling test
istered to the candidate observers. The instructions and scoring
may be devised to familiarize the observer with scaling
techniques supplied by the manufacturer should be followed.
procedures and the experimental task of matching a given
In particular, the illumination level should be kept well within
anchor scale with a perceived difference in stimuli. In an
the photopic range (1000 1x is recommended as a minimum
example (1), the observer was asked to judge the apparent
value) and the spectral quality of the illuminating source
length of a line in comparison to the length of a standard line.
should be near that of north-sky daylight. Failure to identify
The lines were drawn with a heavy black marker on 100 mm
correctly the required number of the plates in the test used
by 150 mm index cards, one line to each of 21 cards. The
should be considered grounds for dismissing the candidate
standard or anchor line, 125 mm long, was assigned a value of
observer.
10 units of length. The other 20 cards had lines of various
6.2.2 Farnsworth-Munsell 100 Hue Test—The Farnsworth-
lengths, both longer and shorter than the anchor line. The
Munsell 100 Hue Test (4) should next be administered to the
anchor and one test card were displayed side by side at a
candidate. While the pseudoisochromatic-plate tests isolate
distance of 0.6 m. Of course, no rulers or other aids were
certain factors of color deficiency, the Farnsworth-Munsel
...