ASTM E3093-20
(Guide)Standard Guide for Structured Small Group Product Evaluations
Standard Guide for Structured Small Group Product Evaluations
SIGNIFICANCE AND USE
4.1 Using best practices for SGPE ensures that decisions made will be based on scientific principles, and the outputs obtained will be more objective than those evaluation sessions conducted without this planning, structure, focus, and best practices. These small group evaluations contrast with more formal product tests that include a prequalified participant sample, hypothesis testing, and statistical analysis. Without best sensory practices and procedures, SGPE may be unstructured, unsystematic, difficult to manage, and may lead to outputs that are unclear, not credible, or ignored. Additionally, the use of proper sensory practices reduces bias among participants with specific sample knowledge or a desire to advance an agenda. This guide provides a framework for conceptualizing, organizing, and executing these SGPE.
4.2 SGPE are used in situations in which formal, hypothesis-driven product evaluations are not required. These include situations in which the decision risk is small or stakeholders feel comfortable in making a decision with the attendant risks, or both. Examples of these situations may include limited availability of samples or other resources, potential patent exposure, or low incidence of target population. The SGPE could be an initial screening step or a precursor test before a more formal product test. In the proper context, SGPE can also be a decision-making tool in and of itself. Using the framework presented here provides a degree of rigor that may be absent when a few people evaluate a product without controlled conditions. A poster presented at the 2009 Pangborn Sensory Science Symposium (1)3 reported the results of a survey on SGPE. 59 % of respondents (N = 92) stated that, at their place of employment, typically, non-sensory professionals organized SGPE. Table 1 summarizes key differences between a typical unstructured product evaluation with a small group not following best practices and an SGPE that follows the best practices outlined...
SCOPE
1.1 This guide covers those occasions in which a small group of individuals (generally between three and ten) with potentially different functional roles and degrees of training in sensory and product evaluation, evaluates a product or series of products for a specific objective, with a pre-identified decision to be made, but without the use of formal hypothesis testing or statistics. In the product testing industry, these are often referred to as “benchings,” “cuttings,” or “bench screenings” or, in the case of food products, “tastings,” “informal tastings,” “team tastings,” or “technical tastings.” In this guide, the term “Small Group Product Evaluation” (SGPE) is used.
1.2 The aim of this guide is to provide best practices to ensure that SGPE are conducted with sufficient rigor to enable the most appropriate decision or to yield the needed learning while considering the risk. Because the participants may be heterogeneous with respect to functional role, knowledge of the issue at hand, sensory sensitivity, and degree of sensory or product evaluation training, the likelihood of agreement on a path forward is not assured. Additionally, participants may have certain biases with respect to the issue to be decided, because of prior knowledge or their role within the organization. These potential derailers can be addressed through proper planning and execution of an SGPE. When SGPE are unstructured, unfocused and experimental error and biases uncontrolled, the outputs of SGPEs do not inform decisions or deliver the desired learning in a scientific manner. The goal of this document is to elevate the practice of small group product evaluations by outlining a structure, defining decision criteria in advance, and providing guidelines for implementation, drawing upon existing sensory theory and methods. Outputs from these SGPE are used to inform decisions and determine next steps including the risks involved with each of these...
General Information
Buy Standard
Standards Content (Sample)
This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E3093 − 20
Standard Guide for
1
Structured Small Group Product Evaluations
This standard is issued under the fixed designation E3093; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope utilizeSPGEtestingandforgoformalsensorytesting.Risksin
doing so must be clearly communicated and agreed to by all
1.1 This guide covers those occasions in which a small
involved parties.
group of individuals (generally between three and ten) with
The proper uses of SGPE are several: to screen variables, to
potentially different functional roles and degrees of training in
establishhypotheses,togaininformationaboutaproductsetor
sensoryandproductevaluation,evaluatesaproductorseriesof
category, to take a course of action where a low risk product
products for a specific objective, with a pre-identified decision
decision is needed or for product learning throughout a
to be made, but without the use of formal hypothesis testing or
statistics. In the product testing industry, these are often development program. In all of these cases, the team must
accepttherisksthatcomewithhavingSGPEoutputstoinform
referred to as “benchings,” “cuttings,” or “bench screenings”
or,inthecaseoffoodproducts,“tastings,”“informaltastings,” a decision. One risk involved in SGPE is missing small
“team tastings,” or “technical tastings.” In this guide, the term differences among products (beta risk), when the goal of the
“Small Group Product Evaluation” (SGPE) is used. evaluation is to find such differences, particularly those differ-
ences that might be important to the consumer. An SGPE
1.2 The aim of this guide is to provide best practices to
failuretofinddifferencesdoesnotmeanthatproductsimilarity
ensure that SGPE are conducted with sufficient rigor to enable
or equivalence is established, since much larger sample sizes
the most appropriate decision or to yield the needed learning
than are common to SPGE’s are required to establish
while considering the risk. Because the participants may be
similarity/equivalence.
heterogeneous with respect to functional role, knowledge of
the issue at hand, sensory sensitivity, and degree of sensory or
1.3 This guide covers the planning and implementation
product evaluation training, the likelihood of agreement on a
processes, including objective setting, method determination,
path forward is not assured. Additionally, participants may
number and types of participants, ballots, sample preparation,
have certain biases with respect to the issue to be decided,
decision criteria, products to be included, review of informa-
because of prior knowledge or their role within the organiza-
tion collected, and management of the post-product evaluation
tion.Thesepotentialderailerscanbeaddressedthroughproper
discussion to arrive at a decision within the small group.
planning and execution of an SGPE. When SGPE are
Documenting and communicating SGPE outputs are also
unstructured, unfocused and experimental error and biases
covered, as well as next steps if a decision cannot be reached.
uncontrolled, the outputs of SGPEs do not inform decisions or
Worked examples across industries including food, household,
deliver the desired learning in a scientific manner. The goal of
and personal care are included. The different types of SGPE
this document is to elevate the practice of small group product
covered include those commonly executed but is not exhaus-
evaluations by outlining a structure, defining decision criteria
tive.
in advance, and providing guidelines for implementation,
drawing upon existing sensory theory and methods. Outputs 1.4 This guide does not cover the use of small group
from these SGPE are used to inform decisions and determine evaluationstopilotresearchortestprotocolsbeforeimplemen-
next steps including the risks involved with each of these. tationinlargerscaletesting.Inaddition,theuseofsmallgroup
SGPEarewidelyused,andwhenproperlyconducted,arean evaluations to substitute for larger evaluations that incorporate
option in the sensory professional’s toolbox. SGPE should be
formal hypothesis testing and statistical analysis or to replace
conducted only when the risks are known, stated, and shared. hedonic testing are neither recommended nor included within
Limitedtimingandresourcesalonearenotadequatereasonsto
thisguide.SGPEthatareregularactivitiesofaqualityfunction
and product reviews that are done for demonstration or
informative purposes with no defi
...
This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E3093 − 19 E3093 − 20
Standard Guide for
1
Structured Small Group Product Evaluations
This standard is issued under the fixed designation E3093; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope
1.1 This guide covers those occasions in which a small group of individuals (generally between three and ten) with potentially
different functional roles and degrees of training in sensory and product evaluation, evaluates a product or series of products for
a specific objective, with a pre-identified decision to be made, but without the use of formal hypothesis testing or statistics. In the
product testing industry, these are often referred to as “benchings,” “cuttings,” or “bench screenings” or, in the case of food
products, “tastings,” “informal tastings,” “team tastings,” or “technical tastings.” In this guide, the term “Small Group Product
Evaluation” (SGPE) is used.
1.2 The aim of this guide is to provide best practices to ensure that SGPE are conducted with sufficient rigor to enable the most
appropriate decision or to yield the needed learning while considering the risk. Because the participants may be heterogeneous with
respect to functional role, knowledge of the issue at hand, sensory sensitivity, and degree of sensory or product evaluation training,
the likelihood of agreement on a path forward is not assured. Additionally, participants may have certain biases with respect to the
issue to be decided, because of prior knowledge or their role within the organization. These potential derailers can be addressed
through proper planning and execution of an SGPE. When SGPE are unstructured, unfocused and experimental error and biases
uncontrolled, the outputs of SGPEs do not inform decisions or deliver the desired learning in a scientific manner. The goal of this
document is to elevate the practice of small group product evaluations by outlining a structure, defining decision criteria in
advance, and providing guidelines for implementation, drawing upon existing sensory theory and methods. Outputs from these
SGPE are used to inform decisions and determine next steps including the risks involved with each of these.
SGPE are widely used, and when properly conducted, are an option in the sensory professional’s toolbox. SGPE should be
conducted only when the risks are known, stated, and shared. Limited timing and resources alone are not adequate reasons to utilize
SPGE testing and forgo formal sensory testing. Risks in doing so must be clearly communicated and agreed to by all involved
parties.
The proper uses of SGPE are several: to screen variables, to establish hypotheses, to gain information about a product set or
category, to take a course of action where a low risk product decision is needed or for product learning throughout a development
program. In all of these cases, the team must accept the risks that come with having SGPE outputs to inform a decision. One risk
involved in SGPE is missing small differences among products (beta risk), when the goal of the evaluation is to find such
differences, particularly those differences that might be important to the consumer. An SGPE failure to find differences does not
mean that product similarity or equivalence is established, since much larger sample sizes than are common to SPGE’s are required
to establish similarity/equivalence.
1.3 This guide covers the planning and implementation processes, including objective setting, method determination, number
and types of participants, ballots, sample preparation, decision criteria, products to be included, review of information collected,
and management of the post-product evaluation discussion to arrive at a decision within the small group. Documenting and
communicating SGPE outputs are also covered, as well as next steps if a decision cannot be reached. Worked examples across
industries including food, household, and personal care are included. The different types of SGPE covered include those commonly
executed but is not exhaustive.
1.4 This guide does not cover the use of small group evaluations to pilot research or test protocols before implementation in
larger scale testing. In addition, the use of small group evaluations to substitute for larger evaluations that incorporate formal
hypothesis testing and statistical a
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.