ASTM E3093-19
(Guide)Standard Guide for Structured Small Group Product Evaluations
Standard Guide for Structured Small Group Product Evaluations
SIGNIFICANCE AND USE
4.1 Using best practices for SGPE ensures that decisions made will be based on scientific principles, and the outputs obtained will be more objective than those evaluation sessions conducted without this planning, structure, focus, and best practices. These small group evaluations contrast with more formal product tests that include a prequalified participant sample, hypothesis testing, and statistical analysis. Without best sensory practices and procedures, SGPE may be unstructured, unsystematic, difficult to manage, and may lead to outputs that are unclear, not credible, or ignored. Additionally, the use of proper sensory practices reduces bias among participants with specific sample knowledge or a desire to advance an agenda. This guide provides a framework for conceptualizing, organizing, and executing these SGPE.
4.2 SGPE are used in situations in which formal, hypothesis-driven product evaluations are not required. These include situations in which the decision risk is small, and/or stakeholders feel comfortable in making a decision with the attendant risks. Examples of these situations may include: limited availability of samples or other resources, potential patent exposure, or low incidence of target population. The SGPE could be an initial screening step or a precursor test before a more formal product test. In the proper context, SGPE can also be a decision-making tool in and of itself. Using the framework presented here provides a degree of rigor that may be absent when a few people evaluate a product without controlled conditions. A poster presented at the 2009 Pangborn Sensory Science Symposium (1) reported the results of a survey on SGPE. Fifty-nine percent of respondents (N = 92) stated that, at their place of employment, typically, non-sensory professionals organized SGPE. Table 1 summarizes key differences between a typical unstructured product evaluation with a small group not following best practices and an SGPE that follows the best practices...
SCOPE
1.1 This guide covers those occasions in which a small group of individuals (generally between three and ten) with potentially different functional roles and degrees of training in sensory and product evaluation, evaluates a product or series of products for a specific objective, with a pre-identified decision to be made, but without the use of formal hypothesis testing or statistics. In the product testing industry, these are often referred to as “benchings,” “cuttings,” or “bench screenings” or, in the case of food products, “tastings,” “informal tastings,” “team tastings,” or “technical tastings.” In this guide, the term “Small Group Product Evaluation” (SGPE) is used.
1.2 The aim of this guide is to provide best practices to ensure that SGPE are conducted with sufficient rigor to enable the most appropriate decision or to yield the needed learning while considering the risk. Because the participants may be heterogeneous with respect to functional role, knowledge of the issue at hand, sensory sensitivity, and degree of sensory or product evaluation training, the likelihood of agreement on a path forward is not assured. Additionally, participants may have certain biases with respect to the issue to be decided, because of prior knowledge or their role within the organization. These potential derailers can be addressed through proper planning and execution of an SGPE. When SGPE are unstructured, unfocused and experimental error and biases uncontrolled, the outputs of SGPEs do not inform decisions or deliver the desired learning in a scientific manner. The goal of this document is to elevate the practice of small group product evaluations by outlining a structure, defining decision criteria in advance, and providing guidelines for implementation, drawing upon existing sensory theory and methods. Outputs from these SGPE are used to inform decisions and determine next steps including the risks involved with each of these...
General Information
Relations
Standards Content (Sample)
NOTICE: This standard has either been superseded and replaced by a new version or withdrawn.
Contact ASTM International (www.astm.org) for the latest information
Designation: E3093 − 19
Standard Guide for
1
Structured Small Group Product Evaluations
This standard is issued under the fixed designation E3093; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope utilize SPGE testing and forgo formal sensory testing. Risks in
doing so must be clearly communicated and agreed to by all
1.1 This guide covers those occasions in which a small
involved parties.
group of individuals (generally between three and ten) with
The proper uses of SGPE are several: to screen variables, to
potentially different functional roles and degrees of training in
establishhypotheses,togaininformationaboutaproductsetor
sensoryandproductevaluation,evaluatesaproductorseriesof
category, to take a course of action where a low risk product
products for a specific objective, with a pre-identified decision
decision is needed or for product learning throughout a
to be made, but without the use of formal hypothesis testing or
development program. In all of these cases, the team must
statistics. In the product testing industry, these are often
accepttherisksthatcomewithhavingSGPEoutputstoinform
referred to as “benchings,” “cuttings,” or “bench screenings”
a decision. One risk involved in SGPE is missing small
or,inthecaseoffoodproducts,“tastings,”“informaltastings,”
“team tastings,” or “technical tastings.” In this guide, the term differences among products (beta risk), when the goal of the
evaluation is to find such differences, particularly those differ-
“Small Group Product Evaluation” (SGPE) is used.
ences that might be important to the consumer. An SGPE
1.2 The aim of this guide is to provide best practices to
failuretofinddifferencesdoesnotmeanthatproductsimilarity
ensure that SGPE are conducted with sufficient rigor to enable
or equivalence is established, since much larger sample sizes
the most appropriate decision or to yield the needed learning
than are common to SPGE’s are required to establish
while considering the risk. Because the participants may be
similarity/equivalence.
heterogeneous with respect to functional role, knowledge of
the issue at hand, sensory sensitivity, and degree of sensory or
1.3 This guide covers the planning and implementation
product evaluation training, the likelihood of agreement on a
processes, including objective setting, method determination,
path forward is not assured. Additionally, participants may
number and types of participants, ballots, sample preparation,
have certain biases with respect to the issue to be decided,
decision criteria, products to be included, review of informa-
because of prior knowledge or their role within the organiza-
tion collected, and management of the post-product evaluation
tion.These potential derailers can be addressed through proper
discussion to arrive at a decision within the small group.
planning and execution of an SGPE. When SGPE are
Documenting and communicating SGPE outputs are also
unstructured, unfocused and experimental error and biases
covered, as well as next steps if a decision cannot be reached.
uncontrolled, the outputs of SGPEs do not inform decisions or
Worked examples across industries including food, household,
deliver the desired learning in a scientific manner. The goal of
and personal care are included. The different types of SGPE
this document is to elevate the practice of small group product
covered include those commonly executed but is not exhaus-
evaluations by outlining a structure, defining decision criteria
tive.
in advance, and providing guidelines for implementation,
1.4 This guide does not cover the use of small group
drawing upon existing sensory theory and methods. Outputs
evaluationstopilotresearchortestprotocolsbeforeimplemen-
from these SGPE are used to inform decisions and determine
tationinlargerscaletesting.Inaddition,theuseofsmallgroup
next steps including the risks involved with each of these.
evaluations to substitute for larger evaluations that incorporate
SGPEarewidelyused,andwhenproperlyconducted,arean
option in the sensory professional’s toolbox. SGPE should be formal hypothesis testing and statistical analysis or to replace
hedonic testing are neither recommended nor included within
conducted only when the risks are known, stated, and shared.
Limitedtimingandresourcesalonearenotadequatereasonsto thisguide.SGPEthatareregularactivitiesofaqualityfunction
and product reviews that are done for demonstration or
informative purposes with no defined decision criteria are also
not covered in this guide.
1
ThistestmethodisunderthejurisdictionofASTMCommitteeE18onSensory
Evaluation and is
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.