ASTM E1499-97(2003)
(Guide)Standard Guide for Selection, Evaluation, and Training of Observers
Standard Guide for Selection, Evaluation, and Training of Observers
SIGNIFICANCE AND USE
The term appearance (see 3.2.1) implies the essential presence of human visual observations. The results of visual observation involve not only the step of observing, accomplished by the eye, but also the inseparable step of interpretation in the brain. Instrumental test methods currently cannot duplicate this second step, and therefore can now only approximate, but not fully measure, appearance. Such instrumental measures of appearance properties are useful only to the extent that they can be correlated to the results of visual observations by observers of the appearance phenomena being evaluated.
Almost invariably, too little attention has been paid to ensuring that the essential visual observations have been properly obtained to provide the basis for correlating visual and instrumental test results. (The only recent book devoted to visual measurements (1)3 has no index entry for observer.)
This guide provides the means for assessing observers, by outlining the requirements and tests for their selection, evaluation, and training. This guide should be useful to all experimenters designing or using visual test methods to provide either direct results in terms of the observation of appearance properties, or the experiments correlating such results with instrumental measures approximating the same appearance properties.
SCOPE
1.1 This guide describes criteria and tests for selecting, evaluating, and training human visual-sensory observers for tasks involving the perception and scaling of properties and phenomena relating to appearance.
1.2 Examples of tests requiring the use of trained observers include but are not limited to those described in the following ASTM standards: on color, Practice D 1535 and Practice E 1360; on color difference, Practice D 1729 and Test Method D 2616; on gloss, Test Method D 4449; on metamerism, Practice D 4086; and on setting tolerances, Practice D 3134.
1.3 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and determine the applicability of regulatory limitations prior to use.
General Information
Relations
Standards Content (Sample)
NOTICE: This standard has either been superseded and replaced by a new version or withdrawn.
Contact ASTM International (www.astm.org) for the latest information.
Designation:E1499–97(Reapproved2003)
Standard Guide for
Selection, Evaluation, and Training of Observers
This standard is issued under the fixed designation E1499; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope 3. Terminology
1.1 This guide describes criteria and tests for selecting, 3.1 Definitions—Definitions of appearance terms in Termi-
evaluating, and training human visual-sensory observers for nology E284 are applicable to this guide.
tasks involving the perception and scaling of properties and 3.2 Definitions of Terms Specific to This Standard:
phenomena relating to appearance. 3.2.1 appearance, n—in psychophysical studies, perception
1.2 Examples of tests requiring the use of trained observers inwhichthespectralandgeometricaspectsofavisualstimulus
include but are not limited to those described in the following are integrated with its illuminating and viewing environment.
ASTM standards: on color, Practice D1535 and Practice 3.2.2 observer, n—one who judges visually, qualitatively or
E1360; on color difference, Practice D1729 and Test Method quantitatively, the content of one or more appearance attributes
D2616; on gloss, Test Method D4449; on metamerism, Prac- in each member of a set of objects or stimuli.
tice D4086; and on setting tolerances, Practice D3134. 3.2.3 scale, v—to assess the content of one or more appear-
1.3 This standard does not purport to address all of the ance attributes in the members of a set of stimuli.
safety concerns, if any, associated with its use. It is the 3.2.3.1 Discussion—Alternatively, scales may be deter-
responsibility of the user of this standard to establish appro- mined by assessing the difference in content of an attribute
priate safety and health practices and determine the applica- with respect to the differences in that attribute among the
bility of regulatory limitations prior to use. members of the set.
2. Referenced Documents 4. Summary of Guide
2.1 ASTM Standards: 4.1 This guide provides descriptions of techniques and tests
D1535 PracticeforSpecifyingColorbytheMunsellSystem for the selection of candidates for observers for use in visual
D1729 Practice for Visual Appraisal of Colors and Color testing, for the evaluation of their capabilities in this field, and
Differences of Diffusely-Illuminated Opaque Materials for their training to enhance these capabilities.
D2616 Test Method for Evaluation of Visual Color Differ- 4.2 Discussion is provided of precautions required for the
ence With a Gray Scale efficientuseofobserversinvisualtests,includingavoidanceof
D3134 PracticeforEstablishingColorandGlossTolerances overtaxing the observers and the control of test variables.
D4086 Practice for Visual Evaluation of Metamerism 4.3 Other considerations of test design, including the num-
D4449 Test Method for Visual Evaluation of Gloss Differ- bers of observers and observations required and the precision
ences Between Surfaces of Similar Appearance of the visual results, are to be covered elsewhere.
E284 Terminology of Appearance
5. Significance and Use
E1360 Practice for Specifying Color by Using the Optical
5.1 The term appearance (see 3.2.1) implies the essential
Society of America Uniform Color Scales System
presence of human visual observations. The results of visual
observation involve not only the step of observing, accom-
This guide is under the jurisdiction of ASTM Committee E12 on Color and
plished by the eye, but also the inseparable step of interpreta-
Appearance and is the direct responsibility of Subcommittee E12.11 on Visual
tion in the brain. Instrumental test methods currently cannot
Methods.
Current edition approved Dec. 1, 2003. Published December 2003. Originally
duplicatethissecondstep,andthereforecannowonlyapproxi-
approved in 1992. Last previous edition approved in 1997 as E1499 – 97. DOI:
mate, but not fully measure, appearance. Such instrumental
10.1520/E1499-97R03.
2 measures of appearance properties are useful only to the extent
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
that they can be correlated to the results of visual observations
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
Standards volume information, refer to the standard’s Document Summary page on
by observers of the appearance phenomena being evaluated.
the ASTM website.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959, United States.
E1499–97 (2003)
fall within the main groupings of observer match points for the light
5.2 Almost invariably, too little attention has been paid to
source used on either Fig. 3 of Ref (5) or the figure of Ref (6). Note,
ensuring that the essential visual observations have been
however, that this criterion is specific to the originalD&H Color Rule
properlyobtainedtoprovidethebasisforcorrelatingvisualand
and does not apply to the currently available MatchPoint Rule.
instrumental test results. (The only recent book devoted to
visual measurements (1) has no index entry for observer.)
6.2.3 Farnsworth-Munsell 100 Hue Test—The Farnsworth-
,
5 8
5.3 This guide provides the means for assessing observers,
Munsell 100 HueTest (7) should next be administered to the
by outlining the requirements and tests for their selection,
candidate. While the pseudoisochromatic-plate tests isolate
evaluation, and training. This guide should be useful to all
certainfactorsofcolordeficiency,theFarnsworth-Munsell100
experimenters designing or using visual test methods to pro-
Hue Test measures color discrimination directly and in detail.
vide either direct results in terms of the observation of
This test was not designed strictly for pass-fail categorization
appearance properties, or the experiments correlating such
of observers but is recommended as an adjunct test for the
results with instrumental measures approximating the same
analysis of color defectives. (It is also useful as an observer
appearance properties.
evaluation test; see 6.3.1.) In the Farnsworth-Munsell 100 Hue
Test, abnormal color vision is indicated by the observer’s
6. Selection and Evaluation of Observers
failure to place the test chips in correct order.The chips consist
6.1 Theprocessusedforselectingobserversdependsagreat
of 85 colored papers varying in hue at approximately constant
deal upon the type of experiment being carried out, but should
valueandchroma,andtheobserver’sfailureisusuallybywide
essentially evaluate the potential capability of the observer to
margins in one or more limited regions of the hue circle. The
execute a series of visual evaluation tasks (2, 3). When these
presenceofsuchabnormalresultsofthetestshouldbegrounds
tasks involve appearance attributes, color or related spectral
for dismissing the candidate observer.
phenomena are often among the task subjects, and if instead
6.3 Visual Acuity and Discrimination Tests—Having deter-
geometric phenomena such as gloss are involved, many of the
mined that the candidate observers have normal color vision, it
same considerations apply. Accordingly, the emphasis in this
is next necessary to test their level of discrimination of small
guide is upon selecting observers for color-related measure-
differences in color or another appearance attribute of interest.
ments. Thus, observers must be screened to rule out those with
6.3.1 Farnsworth-Munsell 100 Hue Test—Use of the
any color- or task-oriented deficiencies.
Farnsworth-Munsell 100 Hue Test as a color-discrimination
6.2 Color Vision Tests (4):
test does not require readministration of the test, but merely
6.2.1 Pseudoisochromatic Plates—As a preliminary color
, ,
4 5 6
reexamination of the test results. For the purposes of assessing
vision test, a pseudoisochromatic-plate test should be
color (more precisely, hue) discrimination, the test results are
administered to the candidate observers. The instructions and
examined for the presence of an approximately constant but
scoring techniques supplied by the manufacturer should be
significant error level in the arrangement of the test chips
followed. In particular, the illumination level should be kept
throughout the hue circle. This may be interpreted as an
well within the photopic range (1000 1x is recommended as a
inability to discriminate the small color differences between
minimum value) and the spectral quality of the illuminating
source should be near that of north-sky daylight. Failure to neighboring chips. While a weakness of this type might, for
example, interfere with an observer’s ability to participate in
identify correctly the required number of the plates in the test
used should be considered grounds for dismissing the candi- threshold scaling experiments, the observer might still be
date observer. competent to perform magnitude scaling of larger differences
6.2.2 Color Rule Test—When the anticipated experiment among specimens.
involves observing in the object mode, the candidate observer
6.3.2 Triangle Test—This test is part of a series known as
,
5 9
should be asked to find a (metameric) match on a Color Rule.
the Japanese Color Aptitude Test. The candidate observers
Astandardlightsourceshouldbeused,similartothatspecified
are shown, one at a time, a series of 20 sets of three colored
in the majority of tests for which the observer is being trained.
chips each. In each set, two of the chips are identical and the
It should be remembered that normal trichromats will report
third is slightly different in color. The observer is asked to
match points that are dependent on age (5), but any abnormal
identify which one is different, the differences being so small
match point should be considered grounds for dismissing the
that there is considerable uncertainty in the judgment.Alower
candidate observer.
than average score in this test indicates that the observer does
not differentiate small differences well.
NOTE 1—An abnormal match point may be considered one that fails to
The boldface numbers in parentheses refer to the list of references at the end of
this standard. D & H Color Rule (no longer available); MatchPoint Rule (available 1990 but
The sole source of supply of the Dvorine Pseudo-Isochromatic Plates, known not equivalent to the D& H Rule), available from Munsell Laboratory, Macbeth
to the committee at this time is The Psychological Corp., Harcourt Brace Division, Kollmorgen Instruments Corp., 405 Little Britain Road, New Windsor,
Jovanovich, 555 Academic Court, San Antonio, TX 78204. NY 12553-6148.
5 8
If you are aware of alternative suppliers, please provide this information to The sole source of supply of the Farnsworth-Munsell 100 Hue Test, known to
ASTM International Headquarters. Your comments will receive careful consider- the committee at this time is Munsell Laboratory, Macbeth Division, Kollmorgen
ation at a meeting of the responsible technical committee, which you may attend. Instruments Corp., 405 Little Britain Rd., New Windson, NY 12553-6148.
6 9
The sole source of supply of the H-R-R Pseudoisochromatic plates, Ishihara The sole source of supply of the Japanese Color Aptitude Test (1994 edition),
Color-Blindness Tests, known to the committee at this time is Richmond Products known to the committee at this time is Japan Color Research Institute, 3-1-19
Inc., 1021 S. Rogers Circle, Suite 6, Boco Raton, FL 33484. Nishiazabu, Minato-ku, Tokyo 106, Japan.
E1499–97 (2003)
6.3.3 HVC (Hue, Value, and Chroma) Color Vision Skill conditions of the scaling experiment in accordance with
5,10
Test—The HVC Color Vision Skill Test is designed to Section 6 cannot be overemphasized. The details of these
assess the ability of the observer to discriminate between training sessions will depend on the experimental design being
specimens having small color differences in hue, value, and used and cannot be stated in general terms here. Examples are
chroma. The test is a general indicator of accuracy in making found in Refs (2) and (3). Only by training the observer under
color matches. It consists of a set of 36 loose specimens for the actual experimental conditions to be used can that observer
one-at-a-time comparison with 36 mounted specimens in a learn exactly what is expected in the task.
prescribed sequence. The specimens are of one of four hues
7.1.1 It is recommended that each observer make a dummy
(blue, red, green, and yellow) with nine specimens each in
set of observations before each observing session. This has
subgroups that vary in hue, value, and chroma around a center
several advantages. The first few observations always exhibit
point. The color differences among the specimens correspond
more“ noise,” as the observer refamiliarizes himself with the
roughly to industrial color matching tolerances.
task. Use of the same sample set for each of these preliminary
6.4 Magnitude Scaling Tests:
observation sets allows consistency to be tested and puts the
6.4.1 Length Estimation—A simple magnitude-scaling test
observer at ease. At least one such set of observations should
may be devised to familiarize the observer with scaling
be one from which the results are discarded. The experimenter
procedures and the experimental task of matching a given
should watch the observer during the first session to be sure
anchor scale with a perceived difference in stimuli. In an
that the instructions for the experiment are understood.
example (2), the observer was asked to judge the apparent
7.1.2 There appears to be virtually no information reported
length of a line in comparison to the length of a standard line.
in Refs (1-3), (9-12) about several important aspects of
The lines were drawn with a heavy black marker on 100 by
observer training, such as how long the training sessions need
150-mmindexcards,onelinetoeachof21cards.Thestandard
tobe,whentheyshouldoccur,whethertheyshouldberepeated
or anchor line, 125 mm long, was assigned a value of 10 units
and at what intervals, and how the experimenter knows when
of length. The other 20 cards had lines of various lengths, both
the observer is adequately trained. A few comments on the
longerandshorterthantheanchorline.Theanchorandonetest
number of observers required in a typical case are given in Ref
card were displayed side by side at a distance of 0.6 m. Of
(9), but this should be considered as part of the experimental
course, no rulers or other aids were allowed. The observer’s
design.
task was to assign a value to the length of each line relative to
7.1.3
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.