ASTM E3093-20

Standard Guide for Structured Small Group Product Evaluations

SIGNIFICANCE AND USE
4.1 Using best practices for SGPE ensures that decisions made will be based on scientific principles, and the outputs obtained will be more objective than those evaluation sessions conducted without this planning, structure, focus, and best practices. These small group evaluations contrast with more formal product tests that include a prequalified participant sample, hypothesis testing, and statistical analysis. Without best sensory practices and procedures, SGPE may be unstructured, unsystematic, difficult to manage, and may lead to outputs that are unclear, not credible, or ignored. Additionally, the use of proper sensory practices reduces bias among participants with specific sample knowledge or a desire to advance an agenda. This guide provides a framework for conceptualizing, organizing, and executing these SGPE.  
4.2 SGPE are used in situations in which formal, hypothesis-driven product evaluations are not required. These include situations in which the decision risk is small or stakeholders feel comfortable in making a decision with the attendant risks, or both. Examples of these situations may include limited availability of samples or other resources, potential patent exposure, or low incidence of target population. The SGPE could be an initial screening step or a precursor test before a more formal product test. In the proper context, SGPE can also be a decision-making tool in and of itself. Using the framework presented here provides a degree of rigor that may be absent when a few people evaluate a product without controlled conditions. A poster presented at the 2009 Pangborn Sensory Science Symposium (1)3 reported the results of a survey on SGPE. 59 % of respondents (N = 92) stated that, at their place of employment, typically, non-sensory professionals organized SGPE. Table 1 summarizes key differences between a typical unstructured product evaluation with a small group not following best practices and an SGPE that follows the best practices outlined...
SCOPE
1.1 This guide covers those occasions in which a small group of individuals (generally between three and ten) with potentially different functional roles and degrees of training in sensory and product evaluation, evaluates a product or series of products for a specific objective, with a pre-identified decision to be made, but without the use of formal hypothesis testing or statistics. In the product testing industry, these are often referred to as “benchings,” “cuttings,” or “bench screenings” or, in the case of food products, “tastings,” “informal tastings,” “team tastings,” or “technical tastings.” In this guide, the term “Small Group Product Evaluation” (SGPE) is used.  
1.2 The aim of this guide is to provide best practices to ensure that SGPE are conducted with sufficient rigor to enable the most appropriate decision or to yield the needed learning while considering the risk. Because the participants may be heterogeneous with respect to functional role, knowledge of the issue at hand, sensory sensitivity, and degree of sensory or product evaluation training, the likelihood of agreement on a path forward is not assured. Additionally, participants may have certain biases with respect to the issue to be decided, because of prior knowledge or their role within the organization. These potential derailers can be addressed through proper planning and execution of an SGPE. When SGPE are unstructured, unfocused and experimental error and biases uncontrolled, the outputs of SGPEs do not inform decisions or deliver the desired learning in a scientific manner. The goal of this document is to elevate the practice of small group product evaluations by outlining a structure, defining decision criteria in advance, and providing guidelines for implementation, drawing upon existing sensory theory and methods. Outputs from these SGPE are used to inform decisions and determine next steps including the risks involved with each of these...

General Information

• Status: Published
• Publication Date: 31-Jan-2020
• ICS: 03.100.01 - Company organization and management in general
• ICS: 19.020 - Test conditions and procedures in general
• Technical Committee: E18 - Sensory Evaluation
• Drafting Committee: E18.05 - Method Applications - Specific Uses or Assessors

Relations

• Replaces: ASTM E3093-19 - Standard Guide for Structured Small Group Product Evaluations
• Effective Date: 01-Feb-2020
• Refers: ASTM E3009-24 - Standard Test Method for Sensory Analysis-Tetrad Test
• Effective Date: 15-Feb-2024
• Refers: ASTM E3009-23a - Standard Test Method for Sensory Analysis-Tetrad Test
• Effective Date: 15-Dec-2023
• Refers: ASTM E1885-18 - Standard Test Method for Sensory Analysis—Triangle Test
• Effective Date: 15-Aug-2018
• Refers: ASTM E2610-18 - Standard Test Method for Sensory Analysis—Duo-Trio Test
• Effective Date: 01-Aug-2018
• Refers: ASTM E2139-05(2018) - Standard Test Method for Same-Different Test
• Effective Date: 01-Aug-2018
• Refers: ASTM E2164-16 - Standard Test Method for Directional Difference Test
• Effective Date: 01-Oct-2016
• Refers: ASTM E3009-15 - Standard Test Method for Sensory Analysis—Tetrad Test
• Effective Date: 01-May-2015
• Refers: ASTM E2610-08(2011) - Standard Test Method for Sensory Analysis—Duo-Trio Test
• Effective Date: 15-Nov-2011
• Refers: ASTM E2139-05(2011) - Standard Test Method for Same-Different Test
• Effective Date: 01-Aug-2011
• Refers: ASTM E1885-04(2011) - Standard Test Method for Sensory Analysis<char: emdash>Triangle Test
• Effective Date: 01-Aug-2011
• Refers: ASTM E2610-08 - Standard Test Method for Sensory Analysis<span class='unicode'>&#x2014;</span>Duo-Trio Test
• Effective Date: 15-Apr-2008
• Refers: ASTM E2164-08 - Standard Test Method for Directional Difference Test
• Effective Date: 01-Mar-2008
• Refers: ASTM E2164-01(2007) - Standard Test Method for Directional Difference Test
• Effective Date: 01-Sep-2007
• Refers: ASTM E2139-05 - Standard Test Method for Same-Different Test
• Effective Date: 01-Nov-2005

Overview

ASTM E3093-20, the "Standard Guide for Structured Small Group Product Evaluations" (SGPE), establishes a best-practice framework for evaluating products in small groups under controlled, structured conditions. This guide, developed by ASTM International, applies to product testing situations where traditional large-scale sensory panels or rigorous statistical testing are unnecessary or impractical. Through defined roles, processes, and decision criteria, ASTM E3093-20 aims to reduce bias, enhance objectivity, and increase the reliability of outcomes in small group product assessments.

SGPEs are commonly used across various industries-including food, household, and personal care-when stakeholder risk is low, resources are limited, or quick, actionable insights are required to inform product development or research directions.

Key Topics

• Best Practices for SGPE: The standard prescribes planning, structuring, and conducting SGPEs to ensure outcomes are scientifically grounded, objective, and actionable. Key elements include clear decision criteria, structured ballots, and balanced sample presentation.
• Participant Selection: SGPEs typically involve 3–10 participants chosen for their relevant sensory acuity, background knowledge, role, or investment in the decision. Selecting diverse participants helps capture a range of perspectives and reduces bias.
• Common SGPE Objectives:
  • Check-in: Verify if a sample meets specific sensory goals.
  • Narrow-down: Eliminate redundant samples or select those with desired profiles.
  • Clarify: Understand sources of external feedback and resolve ambiguities.
  • Describe: Identify and document essential sensory attributes of products.
  • Discover Sensory Dimensions: Explore and define the sensory landscape within a product category.
• Risk Assessment: The guide emphasizes understanding and communicating the risks attached to SGPE decisions, especially since small groups may not detect minor product differences.
• Documentation & Communication: Properly recording the decisions, consensus, and limitations of SGPEs is essential for transparency and for informing next steps in product development.

Applications

ASTM E3093-20 is most valuable in situations where:

• Resource Constraints Exist: When sample availability or testing budgets are limited, and full-scale, hypothesis-driven evaluations are not feasible.
• Low Decision Risk: When the business or development impact of an incorrect decision is small, or further validation can be pursued as needed.
• Screening and Early Development: SGPEs serve as initial filters or “bench screenings” to identify promising candidates for further testing or market introduction.
• Rapid Prototyping: Quick feedback cycles in product innovation can leverage structured SGPEs to accelerate decision-making.
• Exploratory Research: Teams use SGPEs to explore new categories, concepts, or attributes before committing to large-scale testing.
• Special Scenarios: Where sample confidentiality, potential intellectual property exposure, or low target population incidence makes broader testing impractical.

These applications span various sectors, including food and beverage (informal tastings, team cuttings), cosmetics, home care, and technical R&D environments.

Related Standards

ASTM E3093-20 is part of a broader suite of sensory analysis and product evaluation standards, including:

• ASTM E1885: Test Method for Sensory Analysis-Triangle Test
• ASTM E2139: Test Method for Same-Different Test
• ASTM E2164: Test Method for Directional Difference Test
• ASTM E2610: Test Method for Sensory Analysis-Duo-Trio Test
• ASTM E3009: Test Method for Sensory Analysis-Tetrad Test

These methods offer structured approaches for specific sensory and product testing scenarios and can complement SGPEs when more formal hypothesis-driven evaluations are warranted.

---

By adopting ASTM E3093-20, organizations improve the reliability and credibility of small group product evaluations, ensure stakeholder alignment, and support scientifically informed decision-making, even in real-world constraints. This standard is particularly relevant for sensory professionals, product developers, and quality managers seeking efficient, transparent, and robust evaluation processes.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E3093 − 20
Standard Guide for
Structured Small Group Product Evaluations
This standard is issued under the fixed designation E3093; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope utilizeSPGEtestingandforgoformalsensorytesting.Risksin
doing so must be clearly communicated and agreed to by all
1.1 This guide covers those occasions in which a small
involved parties.
group of individuals (generally between three and ten) with
The proper uses of SGPE are several: to screen variables, to
potentially different functional roles and degrees of training in
establishhypotheses,togaininformationaboutaproductsetor
sensoryandproductevaluation,evaluatesaproductorseriesof
category, to take a course of action where a low risk product
products for a specific objective, with a pre-identified decision
decision is needed or for product learning throughout a
to be made, but without the use of formal hypothesis testing or
statistics. In the product testing industry, these are often development program. In all of these cases, the team must
accepttherisksthatcomewithhavingSGPEoutputstoinform
referred to as “benchings,” “cuttings,” or “bench screenings”
or,inthecaseoffoodproducts,“tastings,”“informaltastings,” a decision. One risk involved in SGPE is missing small
“team tastings,” or “technical tastings.” In this guide, the term differences among products (beta risk), when the goal of the
“Small Group Product Evaluation” (SGPE) is used. evaluation is to find such differences, particularly those differ-
ences that might be important to the consumer. An SGPE
1.2 The aim of this guide is to provide best practices to
failuretofinddifferencesdoesnotmeanthatproductsimilarity
ensure that SGPE are conducted with sufficient rigor to enable
or equivalence is established, since much larger sample sizes
the most appropriate decision or to yield the needed learning
than are common to SPGE’s are required to establish
while considering the risk. Because the participants may be
similarity/equivalence.
heterogeneous with respect to functional role, knowledge of
the issue at hand, sensory sensitivity, and degree of sensory or
1.3 This guide covers the planning and implementation
product evaluation training, the likelihood of agreement on a
processes, including objective setting, method determination,
path forward is not assured. Additionally, participants may
number and types of participants, ballots, sample preparation,
have certain biases with respect to the issue to be decided,
decision criteria, products to be included, review of informa-
because of prior knowledge or their role within the organiza-
tion collected, and management of the post-product evaluation
tion.Thesepotentialderailerscanbeaddressedthroughproper
discussion to arrive at a decision within the small group.
planning and execution of an SGPE. When SGPE are
Documenting and communicating SGPE outputs are also
unstructured, unfocused and experimental error and biases
covered, as well as next steps if a decision cannot be reached.
uncontrolled, the outputs of SGPEs do not inform decisions or
Worked examples across industries including food, household,
deliver the desired learning in a scientific manner. The goal of
and personal care are included. The different types of SGPE
this document is to elevate the practice of small group product
covered include those commonly executed but is not exhaus-
evaluations by outlining a structure, defining decision criteria
tive.
in advance, and providing guidelines for implementation,
drawing upon existing sensory theory and methods. Outputs 1.4 This guide does not cover the use of small group
from these SGPE are used to inform decisions and determine evaluationstopilotresearchortestprotocolsbeforeimplemen-
next steps including the risks involved with each of these. tationinlargerscaletesting.Inaddition,theuseofsmallgroup
SGPEarewidelyused,andwhenproperlyconducted,arean evaluations to substitute for larger evaluations that incorporate
option in the sensory professional’s toolbox. SGPE should be
formal hypothesis testing and statistical analysis or to replace
conducted only when the risks are known, stated, and shared. hedonic testing are neither recommended nor included within
Limitedtimingandresourcesalonearenotadequatereasonsto
thisguide.SGPEthatareregularactivitiesofaqualityfunction
and product reviews that are done for demonstration or
informative purposes with no defined decision criteria are also
This guide is under the jurisdiction of ASTM Committee E18 on Sensory
not covered in this guide.
Evaluation and is the direct responsibility of Subcommittee E18.05 on Sensory
Applications--General.
1.5 See 5.2 for a best practice recommendation for the role
Current edition approved Feb. 1, 2020. Published March 2020. Originally
of the sensory professional or trained delegate in the planning,
approved in 2019. Last previous edition approved in 2019 as E3093–19. DOI:
10.1520/E3093-20 designing, conducting, or oversight of structured SGPE.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3093 − 20
TABLE 1 Comparison of Not Best Practice SGPE Versus Best
1.6 This standard does not purport to address all of the
Practice SGPE
safety concerns, if any, associated with its use. It is the
Not Best Practice
responsibility of the user of this standard to establish appro-
Feature Best Practice SGPE
SGPE
priate safety, health, and environmental practices and deter-
# Participants Variable 3–10
mine the applicability of regulatory limitations prior to use.
Rationale for selecting Availability Specified
participants
1.7 This international standard was developed in accor-
Decision criteria None Specified
dance with internationally recognized principles on standard-
Structured ballot No Yes
ization established in the Decision on Principles for the Hedonic response Sometimes No
Product presentation Unsystematic Balanced as possible
Development of International Standards, Guides and Recom-
Independent judgments Sometimes Yes
mendations issued by the World Trade Organization Technical
Discussion Variable Yes
Barriers to Trade (TBT) Committee. Decision is data-driven Sometimes Yes
Output and decision recorded Sometimes Yes
2. Referenced Documents
2.1 ASTM Standards:
4.2 SGPE are used in situations in which formal,
E1885Test Method for Sensory Analysis—Triangle Test
hypothesis-driven product evaluations are not required. These
E2139Test Method for Same-Different Test
include situations in which the decision risk is small or
E2164Test Method for Directional Difference Test
stakeholders feel comfortable in making a decision with the
E2610Test Method for Sensory Analysis—Duo-Trio Test
attendant risks, or both. Examples of these situations may
E3009Test Method for Sensory Analysis—Tetrad Test
include limited availability of samples or other resources,
3. Summary of Guide potential patent exposure, or low incidence of target popula-
tion.TheSGPEcouldbeaninitialscreeningsteporaprecursor
3.1 The aim of this guide is to provide best practices for
test before a more formal product test. In the proper context,
small group product evaluations (SGPE), which are often
SGPEcanalsobeadecision-makingtoolinandofitself.Using
referred to as “benchings,” “cuttings,” or “bench screenings”
the framework presented here provides a degree of rigor that
or,inthecaseoffoodproducts,“tastings,”“informaltastings,”
may be absent when a few people evaluate a product without
“team tasting,” or “technical tastings.” SGPE are used to
controlledconditions.Aposterpresentedatthe2009Pangborn
address learning objectives, make a product decision to con-
Sensory Science Symposium (1) reported the results of a
duct upcoming research, or make a product decision that has
survey on SGPE. 59% of respondents (N = 92) stated that, at
business implications when formal, larger scale testing is not
theirplaceofemployment,typically,non-sensoryprofessionals
required. Best practices are needed to ensure actionable out-
organized SGPE. Table 1 summarizes key differences between
comes or clarity in learning from SGPE when a small number
a typical unstructured product evaluation with a small group
ofpeopleareevaluatingthesamplesandpersonalandpolitical
not following best practices and an SGPE that follows the best
agendas may be in play among multiple stakeholders. This
practices outlined in this guide.
guide outlines a structure for planning and implementing an
SGPE. Outputs from these SGPE will be more actionable and
5. Definition—SGPE
will lead to better informed decision-making or more clarity in
5.1 An SGPE generally consists of three to ten people
learning than when these best practices are not followed.
chosen based on one or more of the following criteria: sensory
acuity, prior knowledge, availability, or investment in the
4. Significance and Use
outcome of the evaluation to make a decision concerning a
4.1 Using best practices for SGPE ensures that decisions
product, products, or a product category. Participants complete
made will be based on scientific principles, and the outputs
a given sensory task that may be quantitative or qualitative in
obtained will be more objective than those evaluation sessions
nature, or both, as instructed by a sensory professional.
conducted without this planning, structure, focus, and best
Responsesarecollectedbythesensoryprofessional,compared
practices. These small group evaluations contrast with more
to pre-defined decision criteria, and a discussion of the re-
formal product tests that include a prequalified participant
sponses and larger context ensues. A consensus decision and
sample, hypothesis testing, and statistical analysis. Without
next steps are reached, recorded, and communicated.
best sensory practices and procedures, SGPE may be
5.2 Sensory Delegate—Planning for an SGPE should be
unstructured, unsystematic, difficult to manage, and may lead
done by the sensory professional. If it is not possible for the
to outputs that are unclear, not credible, or ignored.
sensory professional to plan the SGPE or to attend the
Additionally, the use of proper sensory practices reduces bias
evaluation session, a delegate should be appointed by the
amongparticipantswithspecificsampleknowledgeoradesire
sensory professional. The sensory professional coaches the
to advance an agenda. This guide provides a framework for
delegate in conducting the SGPE. The delegate assumes the
conceptualizing, organizing, and executing these SGPE.
roles and responsibilities of the sensory professional with
particular emphasis on conducting an unbiased evaluation
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM
Standards volume information, refer to the standard’s Document Summary page on The boldface numbers in parentheses refer to a list of references at the end of
the ASTM website. this guide.
E3093 − 20
session.Thedelegateshouldmaintaincontactwiththesensory 6.3.3 In exploratory or discovery research,
professional pre- and post-product evaluation session. If the
6.3.4 When the discussion of product properties yields
organization conducting the SGPE does not employ a sensory
useful information and can provide guidance for next steps,
professional,astaffmember,ormembers,maybetrainedtodo
and
so using the tools and techniques outlined in this guide.
6.3.5 When this guide’s recommendations can be imple-
mented.
6. Issues to Consider Before Planning SGPE
6.4 When Small Group Product Evaluations Are Not
6.1 Effect Size Matters—The processes of planning,
Appropriate—There are situations when SGPE are not appro-
conducting, overseeing, and interpreting the outputs of an
priate:
SGPE described in this guide include the collection of both
6.4.1 When results will be interpreted as formal and taken
quantitative (numerical) and qualitative (verbal or written
out of context by users of the information. While it is not
comments) data. Both of these data types may be used to
possible to know ahead of time whether and how results may
inform the final conclusion. It is expected that there will be
be misused, it is incumbent on those who organize, lead, and
patterns in respondents’ numerical ratings. It is also expected
summarize the evaluation to outline clearly how the recom-
thattherewillbepatternsorthemesintherespondents’product
mendation and next steps were decided and the attendant risks
descriptions. If an effect is large, it is likely detected by even a
accepted.
small group. Thus, the likelihood of missing large differences
6.4.2 When consumer/actual user information is required
amongproductsornotdetectinganintensesensorypropertyin
forthedecision.Ifanorganizationhasnotconductedsufficient
a single product is likely to be small. However, if an effect is
prior research with consumers (for example, drivers of liking,
small,thereisagreaterlikelihoodofitbeingmissedbyasmall
product optimization, or developed sensory based specifica-
group,especiallyconsideringdifferencesinsensorysensitivity,
tions) to be able to relay what consumers’responses are likely
product knowledge, and varied degrees of sensory training.
to be (“we know our consumers don’t like heat/spice and this
6.1.1 ThemeasureofeffectsizewasmadepopularbyJacob
sample is quite hot”).
Cohen in his 1988 book, Statistical Power Analysis for the
6.4.3 When the product or product category is key to the
Behavioral Sciences (2), and a calculation of effect size has
business with respect to revenue, margin, or strategy and the
been used in more quantitative, formal testing. The sensory
decision risk is too high.
professional should consider testing approaches other than
6.4.4 When detailed and more precisely measured product
SGPE if the effect size is small and the risk of missing the
attributes and intensities are needed.
effectislarge,suchaswhendecidingifaproductlotscheduled
6.4.5 Whenstatisticalriskassessmentisneededtosupporta
for release is tainted.
decision.
6.2 Considerations Before Planning and Preparing SGPE—
6.4.6 When the recommendations provided in this guide
Before designing and conducting an SGPE, the sensory pro-
meanttoprovidestructureandrigortotheseevaluationscannot
fessional should make the following assessments to determine
be followed.
if the execution of this structured evaluation is appropriate:
6.4.7 WhentheleaderoftheSGPEcannotissueareportthat
6.2.1 Can the issue or research question be structured and
properly describes the SGPE process and the limitations and
focused enough to be evaluated in a small group?
risks associated with the decision and next steps.
6.2.2 WillanSGPEaddresstheobjectivessoadecisioncan
6.4.8 When the main question to be answered cannot be
be made?
properlyaddressedbecauseofthecapabilitiesorknowledgeof
6.2.3 How will the output be used? Will the output of an
individuals available to participate in the SGPE. For example,
SGPE be used in the proper context keeping in mind its
if the main objective is whether samples can be differentiated
limitations?
and only nondiscriminators are available, an SGPE is not
6.2.4 Have the risks involved in using an SGPE been
appropriate.Additionally,ifthemainquestioniswhetherornot
considered and communicated?
product differences are detectable by consumers or untrained
6.2.5 Canparticipantswiththedesiredcharacteristicsforan
panelists, and only trained panelists are available, an SGPE is
SGPE participation be found and are they available?
not appropriate.
6.3 When to Consider Using SGPE—SGPE are appropriate
6.5 When SGPE MAY be Appropriate—Aspreviouslystated
and may be used in the following situations:
in 6.3 and 6.4, an SGPE is appropriate in low-risk situations
6.3.1 An SGPE can be used for a low-risk decision. A
andmaynotbeappropriatewhenthedecisionriskishigh.The
low-risk decision is one in which an erroneous decision by an
following conditions should be true to use an SGPE in
SGPE will not have a major impact on the issue under
high-decision-risk situations:
consideration. Additionally, if the likelihood of an erroneous
6.5.1 Allstakeholdersareawareoftherisksinherentbothin
decision being identified and reversed in subsequent activities
SGPE in general and those specific risks associated with the
is high, then an SGPE can be considered low risk. The SGPE
situation under consideration,
organizerneedstoprovidethenecessarycaveatsupfrontinthe
6.5.2 All stakeholders are willing to accept the risks noted
planning phase of the SGPE and the written documentation
above, and
summarizing the SGPE results and decision made. The orga-
nization needs to accept these risks. 6.5.3 The alternative to conducting an SGPE is making a
6.3.2 For projects involving screening of products, decision with no product evaluation or input.
E3093 − 20
6.6 Five Most Common Types of SGPE—See Table 2 for a 6.6.4 Describe—Itmaybenecessarytodescribethesensory
summary of the five most common types of SGPE or Rogeaux attributes pertinent to a set of products before taking some
(3) for a slightly different classification scheme. An SGPE subsequent action.
typically addresses, but is not limited to, one of the following 6.6.4.1 If a new product is introduced to the market, it may
five broad objectives:
be prudent for a small group to compare it to the company’s
6.6.1 Check-in—Check-in is conducted to determine if a own products or other competitive products before a more
sample(s) is on track to meet a sensory goal or goals. Post
formal evaluation.
Check-in, the sample(s) may proceed to further testing or the 6.6.4.2 Before testing products with consumers, an SGPE
Check-in may result in a market decision. Examples of
can provide a forum for initial exploration of consumer ballot
Check-in include: development.
6.6.1.1 Determination as to whether sensory properties
6.6.5 Discover Sensory Dimensions—Consumer research
have been maintained between sample development steps
studies often involve exploration of an entire category. The
(bench→pilot plant→manufacturing plant) or after an ingredi-
organization should have all impacted parties agree on what
ent or process change. If the team concludes and accepts that
sensory dimensions constitute category inclusion. Examples
the risk is small in these projects, the step following the SGPE
include:
could be a market decision.Whereas, when the concluded risk
6.6.5.1 Determination of product inclusion for a category
of the outcome of the SGPE in these projects is moderate or
appraisal, and
large, further testing, such as a formal sensory test, would be
6.6.5.2 Determination of product inclusion for competitive
warranted.
assessment.
6.6.1.2 Determination as to whether a product is ready for
6.7 Communicating SGPE Output to a Wider Audience—
larger scale or more formal sensory testing or for a market
The outputs and decisions from an SGPE result from a
decision, such as an introduction,
particular set of people evaluating a specific product set in a
6.6.1.3 Determinationastowhetheraqualityissuehasbeen
specified context. Thus, any communication of these findings
addressed,
should be done with the appropriate caveats.
6.6.1.4 Determinationastowhetheranintendedchangetoa
specific sensory attribute has been addressed, and
7. Five-step Framework for SGPE
6.6.1.5 Determinationastowhetherproductfunctionalityor
There are five steps to planning and executing an SGPE as
sensory attributes deliver as expected.
shown in Fig. 1.
6.6.2 Narrow-down—ANarrow-downsessionisdesignedto
reduce a set of samples with a next step generally of consumer
Step 1—Background and Decision Needed
or sensory testing. The criteria for elimination may be defined
before evaluation. Examples of objectives for Narrow-down 7.1 Gather Information to Define Issue—Before executing
an SGPE, an overall objective should be identified with a
evaluations include:
6.6.2.1 The elimination of redundant sensory profiles, and specified decision criterion. To set an objective that meets the
6.6.2.2 Selecting samples within a desired range of sensory needs of stakeholders, the sensory professional should discuss
profiles. all relevant background with parties representing different
6.6.3 Clarify—At times, product feedback is obtained from functions. Thorough background investigation enables objec-
sourcesexternaltotheprojectteamorcompany.Itiscriticalto tivesettingthatconsidersallpointsofviewofthestakeholders,
understand the feedback before communicating a response or their learning needs, the decisions to be made, and next steps.
initiating product change. Examples in which clarification of The sensory professional should be sensitive to the issues at
feedback is needed may include: hand while remaining unbiased, particularly when an issue has
6.6.3.1 Consumer complaints or praise, become or has the potential to become politically charged or
6.6.3.2 Comments on social media, and when certain individuals have a personal agenda. Relevant
6.6.3.3 Assessing consumer response from formal testing background includes the business situation, business strategy,
such as central location tests, home use tests, or focus groups. product information, and history associated with the issue,
TABLE 2 Summary of Five Common Sensory Objectives and Methods Used in SGPE
Type of Evaluation Sensory Objective Examples of Sensory Method or Task
Check-in Determine if sample achieves the sensory goal Compare to reference sample or description
Narrow-down Eliminate sensory redundancy or choose products with desired Degree of difference products, sorting, qualitative mapping,
sensory profiles, or both and comparison to target sample or description
Clarify Understand external feedback Identify and describe attributes
Describe Identify sensory attributes for consumer ballot or other Identify and describe common sensory attributes of a sample
communication
Discover Sensory Dimensions Discover sensory dimensions relevant to the category Discover and describe sensory attributes of a category and
determine representative products
E3093 − 20
FIG. 1 Steps for Structuring SGPE
E3093 − 20
includinganypriordevelopmentandtestingandfeedbackfrom grouping based on unique attributes, or using more complex
internal or external sources. multivariate sorting techniques such as qualitative multivariate
analysis [consensus sorting of products based on predeter-
7.2 Determine Learning Needed and Decision to be Made—
mined sensory dimensions with qualitatively defined attributes
Once the background and pertinent issues are identified, the
(Beckley) (4)].
sensory professional, in collaboration with the broader set of
(2)Rank the samples in order of intensity for a sensory
stakeholders, can identify the decision to be made or learning
attribute.
needed. The SGPE should provide input to enable making the
7.3.2.2 Ifthegoalistoselectsampleswithinadesiredrange
decision based on independent product evaluations and a
of sensory profiles, examples include:
majority or consensus result. Examples of decisions from the
(1)Eliminate samples that are stronger (weaker) in attri-
five broad objectives include: Check-in: “determine whether
bute X compared with a reference.
the reformulated product is ready to move to the next phase of
(2)Rate the samples for attribute X on a scale.
testing;” Narrow-down: “choose products among a larger
7.3.3 Clarify:
product set to be included in subsequent testing;” Clarify:
7.3.3.1 When feedback needs to be clarified, a reference
“understand consumer complaints to determine next steps;”
Describe: “investigate the attributes to be included on a sample, if available, may be needed to compare against the
sample in question:
consumerballot;”andDiscoverSensoryDimensions:“identify
products that represent the relevant sensory dimensions for a (1)What differences are there between the Reference and
category appraisal.” Sample X?
(2)Which sample has more (or less) of attribute X?
(3)Or, if no reference is available: Would you agree that
Step 2—Translate and Identify Method or Task
this sample has attribute X as described from an external
7.3 Translate the Decision into a Sensory Objective—Once
source?
the decision to be taken is identified and classified within the
7.3.4 Describe—When the goal is to develop a consumer
five broad objectives, the sensory professional should deter-
ballot or communicate sensory properties of a product to
minethespecificsensoryquestionthatneedstobeansweredor
others, gathering open-ended comments is often appropriate:
task that needs to be completed as well as the appropriate
7.3.4.1 For Food Product—Document all sensory attributes
methodologyandparticipantsfortheSGPE.Sensoryquestions
within the dimensions of aroma, appearance, taste, texture,
within the five broad objectives found in Table 2 may include:
feeling factors, and aftertaste.
7.3.1 Check-in—To determine if a sample meets a sensory
7.3.4.2 For Non-food Product—Document sensory and or
goal:
functional attributes as relevant to the products under study.
7.3.1.1 Ifthegoalistodemonstratethatasamplehasorhas
not changed from a prior version or to determine whether a
7.3.5 Discover Sensory Dimensions—When the goal is to
sensory issue has been addressed, the sensory question may develop an understanding of the sensory attributes of an entire
include any of the following:
category, the salient sensory attributes should be explored and
(1)Which sample is more (or less) X? agreed upon as a first step:
(2)Are these two samples the same or different?
(1)What sensory attributes are common to this set of
(3)How are these samples different? samples?
7.3.1.2 Ifthegoalistodeterminewhetherasampleisready (2)What sensory attributes would set a sample apart from
for further testing or market introduction, the sensory profes- the others?
sional needs to determine the criteria that are to be met to
7.4 Identify Sensory Method to Address Objective—The
achieve that goal. Examples include:
choice of a sensory method is based on the nature of the task,
(1)How different is the sample compared to a reference?
theproducts’sensoryprofiles,andtheabilityoftheparticipants
(2)Is the X defect present in this sample?
to complete the tasks involved. Methods range from the
(3)How different is the sample compared with competitor
simplest, such as recording sensory attributes of one or more
X?
samples, to more complex, such as rating, ranking, or sorting
7.3.1.3 Ifthegoalistodeterminewhethersensoryattributes
samples into categories. No matter the method chosen, best
deliver as expected, the question to be asked may be:
practices for sensory evaluation should be implemented: inde-
(1)How X is this sample?
pendent judgments, blind sample coding where appropriate,
(2)If I described this sample as X, would you agree or
consideration of the number of samples to be evaluated in one
disagree?
session, appropriate temperature controls, proper rinsing and
7.3.2 Narrow-down:
evaluation intervals, sample evaluation order, potential for
7.3.2.1 In cases in which a set of samples needs to be
sensory fatigue and carryover, minimization of sensorial
reduced, the sensory question focuses on eliminating samples
distractions, and response recording as instructed. Structured
with redundant, irrelevant, and or inappropriate sensory pro-
ballotsarebestusedfortheseevaluations.SeeSensoryTesting
files. Examples include:
Methods, MNL-26 (5).
(1)Group the samples into sets with similar sensory
7.5 Participants:
attributes.Narrow-downcanbedonebyasortingtaskwhereby
participants individually group samples into sets with similar 7.5.1 Number of Participants—Generally, three to ten
sensory attributes or using check all that apply followed by people participate in an SGPE. If there are more than ten
E3093 − 20
people, the data collection and discussion can become un- how the decision was reached, rather than be informed of the
wieldy. A larger number of people may not allow all partici- decision at a later time.
pantstoexpresstheirpointofview.Participationbyfewerthan (b)Participantsmaybechosenbecausethey do nothavean
threepeoplewillnotallowapatterntoemergeoramajorityto investment in the outcome; in this case, these individuals’
be determined. While odd numbers of participants make it
outputsarelikelytobeunbiasedandtheoutcomecouldbeseen
easier to reach a majority, the absolute small number of as more objective.
participants makes it essential to consider all points of view
7.5.2.4 It is generally advised to invite those individuals
before making conclusions or a decision.
who are invested in the outcome so that they can witness the
7.5.2 Decide Inclusion Criteria for Participants:
objective steps taken to evaluate the product, hear the
7.5.2.1 The choice of participants is important, as their discussion, and see how the decision was reached.
sensory acuity, prior training, and project background knowl- Alternatively,theoutcomecanbesharedwiththoseinvestedin
edgewillaffecttheoutcome.Thecriteriaforincludingspecific
the decision in a separate session, either with or without
individualsintheevaluationsessionareoftenbaseduponallof product evaluation; this however, involves an extra step.
these. However, in reality, participants in the SGPE may be
Additionally,itisrecommendedthatthoseindividualsinvested
included based on more mundane realities such as their in the outcome experience the sensory properties of the
availability or project participation.
products and hear the group discussion as it occurs.
(1) Sensory Activity:
7.5.2.5 There are additional considerations when inviting
(a)Participants may be chosen because they do have
participants to an SGPE. Where possible, the sensory profes-
known sensory acuity for the product attributes under consid-
sional should be present (see 5.2). The sensory professional
eration; in this case, there would be a desire to understand the
may or may not participate in the product evaluation itself per
situation from the perspective of highly sensitive individuals
organization protocol but should instruct the participants as to
(sensitive consumers).Additionally, those with known sensory
thetaskathandandleadthesubsequentdiscussion.Ifthereare
acuity can act as tie-breakers or may diffuse politically tense
individuals within the organization that have the power to
situations with an objective view of the sensory properties.
override or ignore the group decision, these individuals should
(b)Participants may be chosen because they do not have
beinvitedtotheevaluationsession.Thegoalofincludingthese
known sensory acuity for the sensory attributes under consid-
individualsisforthemtohavefirsthandknowledgeofhowthe
eration; in this case, there would be a desire to understand the
results and recommendations were obtained.
situation from the perspective of less sensitive individuals
7.5.2.6 Individuals that cannot follow directions or partici-
(some consumers).
pate in a discussion without emotion should not be included in
7.5.2.2 Participants’ sensory capability and prior training
the evaluation. The sensory professional should learn who the
will affect the results obtained. If it is important to notice a
stakeholders are for the issue at hand and whom to invite or
particularsensorydifference,participantswithknownsensitiv-
exclude from the evaluation session.
ity to that sensory difference should be included. If, on the
7.6 Decide Inclusion Criteria for Products:
other hand, it is important to reflect to the general consumer
population, participants with a wider range of sensory acuity 7.6.1 Products included in the SGPE should meet specified
should be included. criteria. For evaluations to assess whether a sample has met a
sensorygoal(Check-in),thepriorproductversion(s)shouldbe
(2) Prior Knowledge:
included, if available. When the goal of the evaluation session
(a)Participants may be chosen because they do have prior
is to clarify external feedback or decide whether a sample
knowledge or background in the issue at hand; in this case,
contains a defect (Clarify), inclusion of a control product, if
there could be a desire to have these participants address the
issue directly to enable better decision making. available, or in-market products may be useful, including
competitive products as well as products with and without
(b)Participants may be chosen because they do not have
prior knowledge or background in the issue at hand; in this defects, depending on the specifics of the feedback. For
describing sensory attributes, before construction of a con-
case, there would be a desire to make the decision in the
absence of this background. sumer ballot or for communicating with another party
(Describe), the range of products that the ballot or communi-
7.5.2.3 While the sensory professional may have gathered
cation is to cover should be included to ensure complete
all relevant information, as issues arise during the evaluation,
attribute selection. For discovering sensory dimensions (Dis-
participants with specific product knowledge can aid in an-
cover Sensory Dimensions), a range of products sufficient to
swering new product, context, or background questions and
represent the variation in sensory attributes typical for the
may assist in drawing conclusions and reaching consensus.
category should be included.
However, product knowledge known in advance may also
result in biases when evaluating products. In most cases, best
7.6.2 Theinclusionofadditionalproductsmaybehelpfulin
practice is to share the specific product identity or technical
illustratingdifferences/similaritiesbetweenthoseandtheprod-
information after the independent evaluation is completed.
uct(s) in question. These may include manufacturing samples
(3) Investment in the Outcome: from the same or different plants, pilot plant or benchtop
(a)Participants may be chosen because they do have an samples, retail samples, differently aged samples, or competi-
investment in the outcome; in this case, it would be important tive products. In the case in which a sample is being compared
for these individuals to experience personally the SGPE to see to a labeled reference, it may be useful to include a blind
E3093 − 20
control. In all cases, the sensory professional should consider SGPEaredifferentfromthoseofatrainedpanelorappropriate
the impact of adding additional samples to the evaluation set. consumer test. For issues of difference/similarity (typical of
Check-in or Narrow-down objectives) and product inclusion/
Step 3—Decision Criteria and Risk Assessment
exclusion in a larger sample set, risks fall into two general
classes: (1) falsely concluding that differences exist or are
7.7 Identify Decision Criteria:
meaningful, and (2) falsely concluding that differences do not
7.7.1 The decision criteria refer to the measures that will
exist or are not meaningful. For issues of identification of
determine the sensory conclusion. Without these, the output
typical category attributes (typical of Clarify, Describe, and
from an SGPE cannot be effectively used to address the
Discover Sensory Dimensions), there is a risk of missing
decision that needs to be made. The fewer criteria, the better.
relevant attributes. There may be a risk of inclusion of
Having multiple criteria increases the chance that products fall
attributes that confuse consumers when the goal is to describe.
short. Below are examples of decision criteria:
7.9.3 Despite the inability to specify numerical risks, there
(1)Majority rules,
are ways of minimizing risks associated with decision-making
(2)Trends or patterns of responses,
in small group settings. These involve understanding the
(3)Alignment after discussion, and
importance of the product to the organization; review of all
(4)Some combination of the above.
relevant data and background, including the appropriate par-
7.7.2 Decision criteria need to be specified in advance of
ticipants; and identifying in which direction the risk is more
conducting the evaluation. The sensory professional is typi-
acute: missing a real or important difference or identifying a
cally responsible for setting the decision criteria and commu-
false or unimportant difference. The more important the
nicating it to participants. It should be stated that there will be
productinquestionistotheorganizationandthedecisiontobe
timeswhenthedecisioncriteriaarereconsideredbycomments
made in an SGPE, the more important it will be to minimize
or reasoning by one or more participants during the discussion
decision risk.
portion of the product evaluation session. While decision
criteriashouldnotbechangedwithoutarationale,aparticipant
Step 4—Implement
may make a compelling case for a course of action that is not
aligned with the predetermined decision criteria. For example, 7.10 Invite Participants and Assemble Products:
prior relevant research could be recalled or new information 7.10.1 Invite participants that meet criteria determined in
discussed to reframe the issue under consideration. While it is Step 2.
incumbent on the sensory professional leading the SGPE to 7.10.2 Assemble the products that meet criteria in Step 2.
obtain all relevant background, it is recognized that there will
7.11 Create Ballot—The ballot, used to record responses to
be times when pertinent information is revealed during the
the samples, should be simple and focused around the issue at
SGPE.
hand to enable efficient collection and collating of responses.
7.8 Identify Sensory Properties to Consider—It is important Items, such as binary responses, “yes”/“no,” “agree”/
to identify those sensory properties that will impact the “disagree,” and “same”/“different” are good options. These
decision and those that may be ignored in the evaluation binary responses can easily be tabulated by means of counting
session. For example, if the goal is to eliminate products with (5). If using a Check-in evaluation to discriminate between
redundant flavors, appearance and texture attributes may be samples: “determine if Product A is different from/similar to
excluded from the evaluation. Or, if a Check-in is limited to Product B,” “determine if Product C is sweeter than Product
appearance of a food product, it may be unnecessary to taste D,”or“determinewhichamongtheseproductsisclosesttothe
the samples. If aroma in package is the focus of a skin care benchmark,” many discrimination methods are available (for
cream evaluation, it may not be necessary to apply it to the example, Test Methods E1885, E2610, E2139, E3009,or
skin. E2164 as examples of proper discrimination test execution; as
previously stated, statistical hypothesis testing in an SGPE is
7.9 Identify Risks of Erroneous Conclusion and Potential to
not recommended). To narrow down a sample set, which
Minimize:
involves eliminating redundant sensory profiles and extreme
7.9.1 In the preparations for an SGPE, the sensory profes-
outliers, sorting, ranking, or scaling methods may be appropri-
sional should openly discuss risks and limitations with the
ate (6) as well as simple, open-ended questions (“write down
stakeholders or other relevant parties before implementation.
all perceived flavors in these samples”). If attribute scaling is
Both decision risk relevant to the use of company resources in
needed, simple category scales are recommended, minimizing
research and development (R&D) as well as the business risks
thenumberofcategories.Hedonicscalesshouldnotbeusedin
need to be clearly stated and directly discussed.
an SGPE as a small group of employees would not be
7.9.2 Risksarepresentinanytestingorevaluationscenario.
representative of the relevant consumer group. For some
When formal product testing involves the formulation of
objectives(suchasdescribeorDiscoverSensoryDimensions),
hypotheses and statistical inference, risks can be quantitatively
collecting open-ended comments may be the best approach.
specified and addressed through participant sample size and
size of difference to be detected. In the case of an SGPE, the 7.12 Implement—Once the sensory method has been
information collected and decisions made are based on small identified, the sensory professional should set the context for
numbers and formal statistical analysis is not conducted.Thus, the evaluation, deciding how much background and objective
risks cannot be specified numerically. However, while the information is to be shared with participants. This includes a
nature of risks remains the same, the risks associated with an general discussion of the issue at hand and identifying the
E3093 − 20
sensory criteria that are relevant to the sensory question to be productsareallquitespicy/hot”).Theleadershouldhavesome
answered. While too much or inappropriate information may familiarity with the ways in which small groups operate, their
bias the evaluation, the simple instruction to “taste these benefits and disadvantages, group dynamics and methods for
samples” or, in the case of non-food items, “evaluate these coming to agreement and handling disagreements. See Cham-
samples” is generally not sufficient to focus participants’ bers (7) for a review of these topics. While Chambers deals
attention and should be avoided. For example, the sensory mainly with discussions arising from descriptive analysis with
professional should tell the SGPE participants what type of trained panels, the principles apply to other situations where
products they will be tasting and, broadly, what decision will group dynamics are in play.
be made. Further, instructions should indicate how many 7.14.2 The sensory professional needs to be prepared for an
products will be evaluated, in what order, and on what the inconsistentpatternintheresponsesorlackofagreementinthe
SGPE participants should focus. The reveal of the product’s discussion.Lackofagreementmaybedueto:(1)differencesin
identity should be provided at the end of the independent sensory sensitivity among the participants, whereby some
evaluations and after the discussion. If it is necessary to reveal discern differences and others do not, (2) product differences
theproductidentitybeforethestartoftheevaluations,thenthe that are too small to be reliably detected, particularly with a
sensory professional and participants should understand that small group, (3) lack of consensus in the use of sensory
the outputs will be based on more than just the sensory
descriptors, (4) lack of consensus in the determination of the
experience.Postevaluation,moreinformationabouttheobjec- size of differences between products, or (5) use of a heterog-
tive and further discussion of the risks may ensue as partici-
enous group with differences in training or exposure to the
pants focus on both the numerical and qualitative data. It is up products.
to the sensory professional to decide the balance between not
Thereasonforthelackofagreementmaynotbeclear.Inany
providing sufficient information to set the context for the case, the subsequent discussion should include the following
evaluationandprovidinginformationthatmaybiasthepartici-
options: (1) follow up with another SGPE or formal testing if
pants. the hypothesis is that an effect may be present but not
discernible in the SGPE, (2) proceed as if differences were not
7.13 Collect Data and Lead Discussion:
found, if business risk in missing differences is low, or (3)
7.13.1 The sensory professional introduces the task and
proceed as if differences were found, if missing differences
decisioncriteriaorlearningneededtotheparticipants.Specific
results in a high business risk.
evaluation instructions should be provided and standard sen-
7.14.3 Itmaybeusefultoconducttworoundsofevaluation,
sory practices should be used for the evaluation session (6).
oneinwhichsampledifferencesarenotidentifiedandasecond
Participants should be exposed to and evaluate products in the
round in which sample differences have been identified. The
samemanner.Samplesshouldbeevaluatedindependentlywith
first round is without context, while the second round is with
product evaluation order balanced to t
...

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E3093 − 19 E3093 − 20
Standard Guide for
Structured Small Group Product Evaluations
This standard is issued under the fixed designation E3093; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope
1.1 This guide covers those occasions in which a small group of individuals (generally between three and ten) with potentially
different functional roles and degrees of training in sensory and product evaluation, evaluates a product or series of products for
a specific objective, with a pre-identified decision to be made, but without the use of formal hypothesis testing or statistics. In the
product testing industry, these are often referred to as “benchings,” “cuttings,” or “bench screenings” or, in the case of food
products, “tastings,” “informal tastings,” “team tastings,” or “technical tastings.” In this guide, the term “Small Group Product
Evaluation” (SGPE) is used.
1.2 The aim of this guide is to provide best practices to ensure that SGPE are conducted with sufficient rigor to enable the most
appropriate decision or to yield the needed learning while considering the risk. Because the participants may be heterogeneous with
respect to functional role, knowledge of the issue at hand, sensory sensitivity, and degree of sensory or product evaluation training,
the likelihood of agreement on a path forward is not assured. Additionally, participants may have certain biases with respect to the
issue to be decided, because of prior knowledge or their role within the organization. These potential derailers can be addressed
through proper planning and execution of an SGPE. When SGPE are unstructured, unfocused and experimental error and biases
uncontrolled, the outputs of SGPEs do not inform decisions or deliver the desired learning in a scientific manner. The goal of this
document is to elevate the practice of small group product evaluations by outlining a structure, defining decision criteria in
advance, and providing guidelines for implementation, drawing upon existing sensory theory and methods. Outputs from these
SGPE are used to inform decisions and determine next steps including the risks involved with each of these.
SGPE are widely used, and when properly conducted, are an option in the sensory professional’s toolbox. SGPE should be
conducted only when the risks are known, stated, and shared. Limited timing and resources alone are not adequate reasons to utilize
SPGE testing and forgo formal sensory testing. Risks in doing so must be clearly communicated and agreed to by all involved
parties.
The proper uses of SGPE are several: to screen variables, to establish hypotheses, to gain information about a product set or
category, to take a course of action where a low risk product decision is needed or for product learning throughout a development
program. In all of these cases, the team must accept the risks that come with having SGPE outputs to inform a decision. One risk
involved in SGPE is missing small differences among products (beta risk), when the goal of the evaluation is to find such
differences, particularly those differences that might be important to the consumer. An SGPE failure to find differences does not
mean that product similarity or equivalence is established, since much larger sample sizes than are common to SPGE’s are required
to establish similarity/equivalence.
1.3 This guide covers the planning and implementation processes, including objective setting, method determination, number
and types of participants, ballots, sample preparation, decision criteria, products to be included, review of information collected,
and management of the post-product evaluation discussion to arrive at a decision within the small group. Documenting and
communicating SGPE outputs are also covered, as well as next steps if a decision cannot be reached. Worked examples across
industries including food, household, and personal care are included. The different types of SGPE covered include those commonly
executed but is not exhaustive.
1.4 This guide does not cover the use of small group evaluations to pilot research or test protocols before implementation in
larger scale testing. In addition, the use of small group evaluations to substitute for larger evaluations that incorporate formal
hypothesis testing and statistical analysis or to replace hedonic testing are neither recommended nor included within this guide.
SGPE that are regular activities of a quality function and product reviews that are done for demonstration or informative purposes
with no defined decision criteria are also not covered in this guide.
This test method guide is under the jurisdiction of ASTM Committee E18 on Sensory Evaluation and is the direct responsibility of Subcommittee E18.05 on Sensory
Applications--General.
Current edition approved Feb. 1, 2019Feb. 1, 2020. Published May 2019March 2020. Originally approved in 2019. Last previous edition approved in 2019 as E3093 – 19.
DOI: 10.1520/E3093-1910.1520/E3093-20
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3093 − 20
1.5 See 5.2 for a best practice recommendation for the role of the sensory professional or trained delegate in the planning,
designing, conducting, or oversight of structured SGPE.
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.7 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E1885 Test Method for Sensory Analysis—Triangle Test
E2139 Test Method for Same-Different Test
E2164 Test Method for Directional Difference Test
E2610 Test Method for Sensory Analysis—Duo-Trio Test
E3009 Test Method for Sensory Analysis—Tetrad Test
3. Summary of Guide
3.1 The aim of this guide is to provide best practices for small group product evaluations (SGPE), which are often referred to
as “benchings,” “cuttings,” or “bench screenings” or, in the case of food products, “tastings,” “informal tastings,” “team tasting,”
or “technical tastings.” SGPE are used to address learning objectives, make a product decision to conduct upcoming research, or
make a product decision that has business implications when formal, larger scale testing is not required. Best practices are needed
to ensure actionable outcomes or clarity in learning from SGPE when a small number of people are evaluating the samples and
personal and political agendas may be in play among multiple stakeholders. This guide outlines a structure for planning and
implementing an SGPE. Outputs from these SGPE will be more actionable and will lead to better informed decision-making or
more clarity in learning than when these best practices are not followed.
4. Significance and Use
4.1 Using best practices for SGPE ensures that decisions made will be based on scientific principles, and the outputs obtained
will be more objective than those evaluation sessions conducted without this planning, structure, focus, and best practices. These
small group evaluations contrast with more formal product tests that include a prequalified participant sample, hypothesis testing,
and statistical analysis. Without best sensory practices and procedures, SGPE may be unstructured, unsystematic, difficult to
manage, and may lead to outputs that are unclear, not credible, or ignored. Additionally, the use of proper sensory practices reduces
bias among participants with specific sample knowledge or a desire to advance an agenda. This guide provides a framework for
conceptualizing, organizing, and executing these SGPE.
4.2 SGPE are used in situations in which formal, hypothesis-driven product evaluations are not required. These include
situations in which the decision risk is small, and/orsmall or stakeholders feel comfortable in making a decision with the attendant
risks. risks, or both. Examples of these situations may include:include limited availability of samples or other resources, potential
patent exposure, or low incidence of target population. The SGPE could be an initial screening step or a precursor test before a
more formal product test. In the proper context, SGPE can also be a decision-making tool in and of itself. Using the framework
presented here provides a degree of rigor that may be absent when a few people evaluate a product without controlled conditions.
A poster presented at the 2009 Pangborn Sensory Science Symposium (1) reported the results of a survey on SGPE. Fifty-nine
percent 59 % of respondents (N = 92) stated that, at their place of employment, typically, non-sensory professionals organized
SGPE. Table 1 summarizes key differences between a typical unstructured product evaluation with a small group not following
best practices and an SGPE that follows the best practices outlined in this guide.
5. Definition: SGPEDefinition—SGPE
5.1 An SGPE generally consists of three to ten people chosen based on one or more of the following criteria: sensory acuity,
prior knowledge, availability, or investment in the outcome of the evaluation to make a decision concerning a product, products,
or a product category. Participants complete a given sensory task that may be quantitative and/oror qualitative in nature, or both,
as instructed by a sensory professional. Responses are collected by the sensory professional, compared to pre-defined decision
criteria, and a discussion of the responses and larger context ensues. A consensus decision and next steps are reached, recorded,
and communicated.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
The boldface numbers in parentheses refer to a list of references at the end of this guide.
E3093 − 20
TABLE 1 Comparison of Not Best Practice SGPE versus Best
Practice SGPE
Not Best Practice Best Practice
Feature
SGPE SGPE
# Participants Variable 3–10
Rationale for selecting Availability Specified
participants
Decision criteria None Specified
Structured ballot No Yes
Hedonic response Sometimes No
Product presentation Unsystematic Balanced
as possible
Independent judgments Sometimes Yes
Discussion Variable Yes
Decision is data-driven Sometimes Yes
Output and decision Sometimes Yes
recorded
TABLE 1 Comparison of Not Best Practice SGPE Versus Best
Practice SGPE
Not Best Practice Best Practice
Feature
SGPE SGPE
# Participants Variable 3–10
Rationale for selecting Availability Specified
participants
Decision criteria None Specified
Structured ballot No Yes
Hedonic response Sometimes No
Product presentation Unsystematic Balanced
as possible
Independent judgments Sometimes Yes
Discussion Variable Yes
Decision is data-driven Sometimes Yes
Output and decision Sometimes Yes
recorded
5.2 Sensory Delegate—Planning for an SGPE should be done by the sensory professional. If it is not possible for the sensory
professional to plan the SGPE or to attend the evaluation session, a delegate should be appointed by the sensory professional. The
sensory professional coaches the delegate in conducting the SGPE. The delegate assumes the roles and responsibilities of the
sensory professional with particular emphasis on conducting an unbiased evaluation session. The delegate should maintain contact
with the sensory professional pre- and post-product evaluation session. If the organization conducting the SGPE does not employ
a sensory professional, a staff member, or members, may be trained to do so using the tools and techniques outlined in this guide.
6. Issues to Consider Before Planning an SGPE
6.1 Effect Size Matters—The processes of planning, conducting, overseeing, and interpreting the outputs of an SGPE described
in this guide include the collection of both quantitative (numerical) and qualitative (verbal or written comments) data. Both of these
data types may be used to inform the final conclusion. It is expected that there will be patterns in respondents’ numerical ratings.
It is also expected that there will be patterns or themes in the respondents’ product descriptions. If an effect is large, it is likely
detected by even a small group. Thus, the likelihood of missing large differences among products or not detecting an intense
sensory property in a single product is likely to be small. However, if an effect is small, there is a greater likelihood of it being
missed by a small group, especially considering differences in sensory sensitivity, product knowledge, and varied degrees of
sensory training.
6.1.1 The measure of effect size was made popular by Jacob Cohen in his 1988 book, Statistical Power Analysis for the
Behavioral Sciences(2), and a calculation of effect size has been used in more quantitative, formal testing. The sensory professional
should consider testing approaches other than SGPE if the effect size is small and the risk of missing the effect is large, such as
when deciding if a product lot scheduled for release is tainted.
6.2 Considerations Before Planning and Preparing an SGPE—Before designing and conducting an SGPE, the sensory
professional should make the following assessments to determine if the execution of this structured evaluation is appropriate:
6.2.1 Can the issue or research question be structured and focused enough to be evaluated in a small group?
6.2.2 Will an SGPE address the objectives so a decision can be made?
6.2.3 How will the output be used? Will the output of an SGPE be used in the proper context keeping in mind its limitations?
6.2.4 Have the risks involved in using an SGPE been considered and communicated?
6.2.5 Can participants with the desired characteristics for an SGPE participation be found and are they available?
6.3 When to Consider Using SGPE—SGPE are appropriate and may be used in the following situations:
E3093 − 20
6.3.1 An SGPE can be used for a low-risk decision. A low-risk decision is one in which an erroneous decision by an SGPE will
not have a major impact on the issue under consideration. Additionally, if the likelihood of an erroneous decision being identified
and reversed in subsequent activities is high, then an SGPE can be considered low risk. The SGPE organizer needs to provide the
necessary caveats upfront in the planning phase of the SGPE and the written documentation summarizing the SGPE results and
decision made. The organization needs to accept these risks.
6.3.2 For projects involving screening of products;products,
6.3.3 In exploratory or discovery research; research,
6.3.4 When the discussion of product properties yields useful information and can provide guidance for next steps;steps, and
6.3.5 When this guide’s recommendations can be implemented.
6.4 When Small Group Product Evaluations Are Not Appropriate—There are situations when SGPE are not appropriate:
6.4.1 When results will be interpreted as formal and taken out of context by users of the information. While it is not possible
to know ahead of time whether and how results may be misused, it is incumbent on those who organize, lead, and summarize the
evaluation to outline clearly how the recommendation and next steps were decided and the attendant risks accepted.
6.4.2 When Consumer/Actual User Information Is Required for the Decision—When consumer/actual user information is
required for the decision. If an organization has not conducted sufficient prior research with consumers (for example, drivers of
liking, product optimization, or developed sensory based specifications) to be able to relay what consumers’ responses are likely
to be (“we know our consumers don’t like heat/spice and this sample is quite hot”).
6.4.3 When the product or product category is key to the business with respect to revenue, margin, or strategy and the decision
risk is too high.
6.4.4 When detailed and more precisely measured product attributes and intensities are needed.
6.4.5 When statistical risk assessment is needed to support a decision.
6.4.6 When the recommendations provided in this guide meant to provide structure and rigor to these evaluations cannot be
followed.
6.4.7 When the leader of the SGPE cannot issue a report that properly describes the SGPE process and the limitations and risks
associated with the decision and next steps.
6.4.8 When the main question to be answered cannot be properly addressed because of the capabilities or knowledge of
individuals available to participate in the SGPE. For example, if the main objective is whether samples can be differentiated and
only nondiscriminators are available, an SGPE is not appropriate. Additionally, if the main question is whether or not product
differences are detectable by consumers or untrained panelists, and only trained panelists are available, an SGPE is not appropriate.
6.5 When an SGPE MAY be Appropriate—As previously stated in 6.3 and 6.4, an SGPE is appropriate in low-risk situations
and may not be appropriate when the decision risk is high. The following conditions should be true to use an SGPE in
high-decision-risk situations:
6.5.1 All stakeholders are aware of the risks inherent both in SGPE in general and those specific risks associated with the
situation under consideration;consideration,
6.5.2 All stakeholders are willing to accept the risks noted above;above, and
6.5.3 The alternative to conducting an SGPE is making a decision with no product evaluation or input.
6.6 Five Most Common Types of SGPE—See Table 2 for a summary of the five most common types of SGPE or Rogeaux (3)
for a slightly different classification scheme. An SGPE typically addresses, but is not limited to, one of the following five broad
objectives:
TABLE 2 Summary of Five Common Sensory Objectives and Methods Used in SGPE
Type of Evaluation Sensory Objective Examples of Sensory Method or Task
Check-in Determine if sample achieves the sensory goal Compare to reference sample or description
Check-in Determine if sample achieves the sensory goal Compare to reference sample or description
Narrow-down Eliminate sensory redundancy or choose products with desired Degree of difference products, sorting, qualitative mapping,
sensory profiles or both and comparison to target sample or description
Narrow-down Eliminate sensory redundancy or choose products with desired Degree of difference products, sorting, qualitative mapping,
sensory profiles, or both and comparison to target sample or description
Clarify Understand external feedback Identify and describe attributes
Clarify Understand external feedback Identify and describe attributes
Describe Identify sensory attributes for consumer ballot or other Identify and describe common sensory attributes of a sample
communication
Describe Identify sensory attributes for consumer ballot or other Identify and describe common sensory attributes of a sample
communication
Discover Sensory Discover sensory dimensions relevant to the category Discover and describe sensory attributes of a category and
Dimensions determine representative products
Discover Sensory Discover sensory dimensions relevant to the category Discover and describe sensory attributes of a category and
Dimensions determine representative products
E3093 − 20
6.6.1 Check-in—Check-in is conducted to determine if a sample(s) is on track to meet a sensory goal or goals. Post Check-in,
the sample(s) may proceed to further testing or the Check-in may result in a market decision. Examples of Check-in include:
6.6.1.1 Determination as to whether sensory properties
have been maintained between sample development steps
(bench→pilot plant→manufacturing plant) or after an ingredient or process change. If the team concludes and accepts that the risk
is small in these projects, the step following the SGPE could be a market decision. Whereas, when the concluded risk of the
outcome of the SGPE in these projects is moderate or large, further testing, such as a formal sensory test, would be warranted.
6.6.1.2 Determination as to whether a product is ready for larger scale or more formal sensory testing or for a market decision,
such as an introduction;introduction,
6.6.1.3 Determination as to whether a quality issue has been addressed;addressed,
6.6.1.4 Determination as to whether an intended change to a specific sensory attribute has been addressed;addressed, and
6.6.1.5 Determination as to whether product functionality or sensory attributes deliver as expected.
6.6.2 Narrow-down—A Narrow-down session is designed to reduce a set of samples with a next step generally of consumer or
sensory testing. The criteria for elimination may be defined before evaluation. Examples of objectives for Narrow-down
evaluations include:
6.6.2.1 The elimination of redundant sensory profiles;profiles, and
6.6.2.2 Selecting samples within a desired range of sensory profiles.
6.6.3 Clarify—At times, product feedback is obtained from sources external to the project team or company. It is critical to
understand the feedback before communicating a response or initiating product change. Examples in which clarification of
feedback is needed may include:
6.6.3.1 Consumer complaints or praise;praise,
6.6.3.2 Comments on social media;media, and
6.6.3.3 Assessing consumer response from formal testing such as central location tests, home use tests, or focus groups.
6.6.4 Describe—It may be necessary to describe the sensory attributes pertinent to a set of products before taking some
subsequent action.
6.6.4.1 If a new product is introduced to the market, it may be prudent for a small group to compare it to the company’s own
products or other competitive products before a more formal evaluation.
6.6.4.2 Before testing products with consumers, an SGPE can provide a forum for initial exploration of consumer ballot
development.
6.6.5 Discover Sensory Dimensions—Consumer research studies often involve exploration of an entire category. The
organization should have all impacted parties agree on what sensory dimensions constitute category inclusion. Examples include:
6.6.5.1 Determination of product inclusion for a category appraisal;appraisal, and
6.6.5.2 Determination of product inclusion for competitive assessment.
6.7 Communicating an SGPE Output to a Wider Audience—The outputs and decisions from an SGPE result from a particular
set of people evaluating a specific product set in a specified context. Thus, any communication of these findings should be done
with the appropriate caveats.
7. Five-StepFive-step Framework for an SGPE
There are five steps to planning and executing an SGPE as shown in Fig. 1.
Step 1—Background and Decision Needed
7.1 Gather Information to Define Issue—Before executing an SGPE, an overall objective should be identified with a specified
decision criterion. To set an objective that meets the needs of stakeholders, the sensory professional should discuss all relevant
background with parties representing different functions. Thorough background investigation enables objective setting that
considers all points of view of the stakeholders, their learning needs, the decisions to be made, and next steps. The sensory
professional should be sensitive to the issues at hand while remaining unbiased, particularly when an issue has become or has the
potential to become politically charged or when certain individuals have a personal agenda. Relevant background includes the
business situation, business strategy, product information, and history associated with the issue, including any prior development
and testing and feedback from internal or external sources.
7.2 Determine Learning Needed and Decision to be Made—Once the background and pertinent issues are identified, the sensory
professional, in collaboration with the broader set of stakeholders, can identify the decision to be made or learning needed. The
SGPE should provide input to enable making the decision based on independent product evaluations and a majority or consensus
result. Examples of decisions from the five broad objectives include: Check-in: “determine whether the reformulated product is
ready to move to the next phase of testing;” Narrow-down: “choose products among a larger product set to be included in
subsequent testing;” Clarify: “understand consumer complaints to determine next steps;” Describe: “investigate the attributes to
be included on a consumer ballot;” and Discover Sensory Dimensions: “identify products that represent the relevant sensory
dimensions for a category appraisal.”
E3093 − 20
FIG. 1 Steps for Structuring an SGPE
E3093 − 20
Step 2—Translate and Identify Method or Task
7.3 Translate the Decision into a Sensory Objective—Once the decision to be taken is identified and classified within the five
broad objectives, the sensory professional should determine the specific sensory question that needs to be answered or task that
needs to be completed as well as the appropriate methodology and participants for the SGPE. Sensory questions within the five
broad objectives found in Table 2 may include:
7.3.1 Check-inCheck-in——To determine if a sample meets a sensory goal:
7.3.1.1 If the goal is to demonstrate that a sample has or has not changed from a prior version or to determine whether a sensory
issue has been addressed, the sensory question may include any of the following:
(1) Which sample is more (or less) X?
(2) Are these two samples the same or different?
(3) How are these samples different?
7.3.1.2 If the goal is to determine whether a sample is ready for further testing or market introduction, the sensory professional
needs to determine the criteria that are to be met to achieve that goal. Examples include:
(1) How different is the sample compared to a reference?
(2) Is the X defect present in this sample?
(3) How different is the sample compared with competitor X?
7.3.1.3 If the goal is to determine whether sensory attributes deliver as expected, the question to be asked may be:
(1) How X is this sample?
(2) If I described this sample as X, would you agree or disagree?
7.3.2 Narrow-down:
7.3.2.1 In cases in which a set of samples needs to be reduced, the sensory question focuses on eliminating samples with
redundant, irrelevant, and or inappropriate sensory profiles. Examples include:
(1) Group the samples into sets with similar sensory attributes. Narrow-down can be done by a sorting task whereby
participants individually group samples into sets with similar sensory attributes or using check all that apply followed by grouping
based on unique attributes, or using more complex multivariate sorting techniques such as qualitative multivariate analysis
[consensus sorting of products based on predetermined sensory dimensions with qualitatively defined attributes (Beckley) (4)].
(2) Rank the samples in order of intensity for a sensory attribute.
7.3.2.2 If the goal is to select samples within a desired range of sensory profiles, examples include:
(1) Eliminate samples that are stronger (weaker) in attribute X compared with a reference.
(2) Rate the samples for attribute X on a scale.
7.3.3 Clarify:
7.3.3.1 When feedback needs to be clarified, a reference sample, if available, may be needed to compare against the sample in
question:
(1) What differences are there between the Reference and Sample X?
(2) Which sample has more (or less) of attribute X?
(3) Or, if no reference is available: Would you agree that this sample has attribute X as described from an external source?
7.3.4 DescribeDescribe——When the goal is to develop a consumer ballot or communicate sensory properties of a product to
others, gathering open-ended comments is often appropriate:
7.3.4.1 For a Food Product—Document all sensory attributes within the dimensions of aroma, appearance, taste, texture, feeling
factors, and aftertaste.
7.3.4.2 For a Non-food Product—Document sensory and or functional attributes as relevant to the products under study.
7.3.5 Discover Sensory DimensionsDimensions——When the goal is to develop an understanding of the sensory attributes of
an entire category, the salient sensory attributes should be explored and agreed upon as a first step:
(1) What sensory attributes are common to this set of samples?
(2) What sensory attributes would set a sample apart from the others?
7.4 Identify Sensory Method to Address Objective—The choice of a sensory method is based on the nature of the task, the
products’ sensory profiles, and the ability of the participants to complete the tasks involved. Methods range from the simplest, such
as recording sensory attributes of one or more samples, to more complex, such as rating, ranking, or sorting samples into
categories. No matter the method chosen, best practices for sensory evaluation should be implemented: independent judgments,
blind sample coding where appropriate, consideration of the number of samples to be evaluated in one session, appropriate
temperature controls, proper rinsing and evaluation intervals, sample evaluation order, potential for sensory fatigue and carryover,
minimization of sensorial distractions, and response recording as instructed. Structured ballots are best used for these evaluations.
See Sensory Testing Methods, MNL-26 (5).
7.5 Participants:
7.5.1 Number of Participants—Generally, three to ten people participate in an SGPE. If there are more than ten people, the data
collection and discussion can become unwieldy. A larger number of people may not allow all participants to express their point
of view. Participation by fewer than three people will not allow a pattern to emerge or a majority to be determined. While odd
E3093 − 20
numbers of participants make it easier to reach a majority, the absolute small number of participants makes it essential to consider
all points of view before making conclusions or a decision.
7.5.2 Decide Inclusion Criteria for Participants:
7.5.2.1 The choice of participants is important, as their sensory acuity, prior training, and project background knowledge will
affect the outcome. The criteria for including specific individuals in the evaluation session are often based upon all of these.
However, in reality, participants in the SGPE may be included based on more mundane realities such as their availability or project
participation.
(1) Sensory Activity:
(a) Participants may be chosen because they do have known sensory acuity for the product attributes under consideration; in
this case, there would be a desire to understand the situation from the perspective of highly sensitive individuals (sensitive
consumers). Additionally, those with known sensory acuity can act as tie-breakers or may diffuse politically tense situations with
an objective view of the sensory properties.
(b) Participants may be chosen because they do not have known sensory acuity for the sensory attributes under consideration;
in this case, there would be a desire to understand the situation from the perspective of less sensitive individuals (some consumers).
7.5.2.2 Participants’ sensory capability and prior training will affect the results obtained. If it is important to notice a particular
sensory difference, participants with known sensitivity to that sensory difference should be included. If, on the other hand, it is
important to reflect to the general consumer population, participants with a wider range of sensory acuity should be included.
(2) Prior Knowledge:
(a) Participants may be chosen because they do have prior knowledge or background in the issue at hand; in this case, there
could be a desire to have these participants address the issue directly to enable better decision making.
(b) Participants may be chosen because they do not have prior knowledge or background in the issue at hand; in this case, there
would be a desire to make the decision in the absence of this background.
7.5.2.3 While the sensory professional may have gathered all relevant information, as issues arise during the evaluation,
participants with specific product knowledge can aid in answering new product, context, or background questions and may assist
in drawing conclusions and reaching consensus. However, product knowledge known in advance may also result in biases when
evaluating products. In most cases, best practice is to share the specific product identity or technical information after the
independent evaluation is completed.
(3) Investment in the Outcome:
(a) Participants may be chosen because they do have an investment in the outcome; in this case, it would be important for these
individuals to experience personally the SGPE to see how the decision was reached, rather than be informed of the decision at a
later time.
(b) Participants may be chosen because they do not have an investment in the outcome; in this case, these individuals’ outputs
are likely to be unbiased and the outcome could be seen as more objective.
7.5.2.4 It is generally advised to invite those individuals who are invested in the outcome so that they can witness the objective
steps taken to evaluate the product, hear the discussion, and see how the decision was reached. Alternatively, the outcome can be
shared with those invested in the decision in a separate session, either with or without product evaluation; this however, involves
an extra step. Additionally, it is recommended that those individuals invested in the outcome experience the sensory properties of
the products and hear the group discussion as it occurs.
7.5.2.5 There are additional considerations when inviting participants to an SGPE. Where possible, the sensory professional
should be present (see 5.2). The sensory professional may or may not participate in the product evaluation itself per organization
protocol but should instruct the participants as to the task at hand and lead the subsequent discussion. If there are individuals within
the organization that have the power to override or ignore the group decision, these individuals should be invited to the evaluation
session. The goal of including these individuals is for them to have firsthand knowledge of how the results and recommendations
were obtained.
7.5.2.6 Individuals that cannot follow directions or participate in a discussion without emotion should not be included in the
evaluation. The sensory professional should learn who the stakeholders are for the issue at hand and whom to invite or exclude
from the evaluation session.
7.6 Decide Inclusion Criteria for Products:
7.6.1 Products included in the SGPE should meet specified criteria. For evaluations to assess whether a sample has met a
sensory goal (Check-in), the prior product version(s) should be included, if available. When the goal of the evaluation session is
to clarify external feedback or decide whether a sample contains a defect (Clarify), inclusion of a control product, if available, or
in-market products may be useful, including competitive products as well as products with and without defects, depending on the
specifics of the feedback. For describing sensory attributes, before construction of a consumer ballot or for communicating with
another party (Describe), the range of products that the ballot or communication is to cover should be included to ensure complete
attribute selection. For discovering sensory dimensions (Discover Sensory Dimensions), a range of products sufficient to represent
the variation in sensory attributes typical for the category should be included.
7.6.2 The inclusion of additional products may be helpful in illustrating differences/similarities between those and the product(s)
in question. These may include manufacturing samples from the same or different plants, pilot plant or benchtop samples, retail
E3093 − 20
samples, differently aged samples, or competitive products. In the case in which a sample is being compared to a labeled reference,
it may be useful to include a blind control. In all cases, the sensory professional should consider the impact of adding additional
samples to the evaluation set.
Step 3—Decision Criteria and Risk Assessment
7.7 Identify Decision Criteria:
7.7.1 The decision criteria refer to the measures that will determine the sensory conclusion. Without these, the output from an
SGPE cannot be effectively used to address the decision that needs to be made. The fewer criteria, the better. Having multiple
criteria increases the chance that products fall short. Below are examples of decision criteria:
(1) Majority rules,
(2) Trends or patterns of responses,
(3) Alignment after discussion, and
(4) Some combination of the above.
7.7.2 Decision criteria need to be specified in advance of conducting the evaluation. The sensory professional is typically
responsible for setting the decision criteria and communicating it to participants. It should be stated that there will be times when
the decision criteria are reconsidered by comments or reasoning by one or more participants during the discussion portion of the
product evaluation session. While decision criteria should not be changed without a rationale, a participant may make a compelling
case for a course of action that is not aligned with the predetermined decision criteria. For example, prior relevant research could
be recalled or new information discussed to reframe the issue under consideration. While it is incumbent on the sensory
professional leading the SGPE to obtain all relevant background, it is recognized that there will be times when pertinent
information is revealed during the SGPE.
7.8 Identify Sensory Properties to Consider—It is important to identify those sensory properties that will impact the decision
and those that may be ignored in the evaluation session. For example, if the goal is to eliminate products with redundant flavors,
appearance and texture attributes may be excluded from the evaluation. Or, if a Check-in is limited to appearance of a food product,
it may be unnecessary to taste the samples. If aroma in package is the focus of a skin care cream evaluation, it may not be necessary
to apply it to the skin.
7.9 Identify Risks of Erroneous Conclusion and Potential to Minimize:
7.9.1 In the preparations for an SGPE, the sensory professional should openly discuss risks and limitations with the stakeholders
or other relevant parties before implementation. Both decision risk relevant to the use of company resources in research and
development (R&D) as well as the business risks need to be clearly stated and directly discussed.
7.9.2 Risks are present in any testing or evaluation scenario. When formal product testing involves the formulation of
hypotheses and statistical inference, risks can be quantitatively specified and addressed through participant sample size and size
of difference to be detected. In the case of an SGPE, the information collected and decisions made are based on small numbers
and formal statistical analysis is not conducted. Thus, risks cannot be specified numerically. However, while the nature of risks
remains the same, the risks associated with an SGPE are different from those of a trained panel or appropriate consumer test. For
issues of difference/similarity (typical of Check-in or Narrow-down objectives) and product inclusion/exclusion in a larger sample
set, risks fall into two general classes: (1) falsely concluding that differences exist or are meaningful, and (2) falsely concluding
that differences do not exist or are not meaningful. For issues of identification of typical category attributes (typical of Clarify,
Describe, and Discover Sensory Dimensions), there is a risk of missing relevant attributes. There may be a risk of inclusion of
attributes that confuse consumers when the goal is to describe.
7.9.3 Despite the inability to specify numerical risks, there are ways of minimizing risks associated with decision-making in
small group settings. These involve understanding the importance of the product to the organization; review of all relevant data
and background, including the appropriate participants; and identifying in which direction the risk is more acute: missing a real
or important difference or identifying a false or unimportant difference. The more important the product in question is to the
organization and the decision to be made in an SGPE, the more important it will be to minimize decision risk.
Step 4—Implement
7.10 Invite Participants and Assemble Products:
7.10.1 Invite participants that meet criteria determined in Step 2.
7.10.2 Assemble the products that meet criteria in Step 2.
7.11 Create Ballot—The ballot, used to record responses to the samples, should be simple and focused around the issue at hand
to enable efficient collection and collating of responses. Items, such as binary responses, “yes”/“no” “agree”/“disagree”,
“yes”/“no,” “agree”/“disagree,” and “same”/“different” are good options. These binary responses can easily be tabulated by means
of counting (5). If using a Check-in evaluation to discriminate between samples: “determine if Product A is different from/similar
to Product B,” “determine if Product C is sweeter than Product D,” or “determine which among these products is closest to the
benchmark,” many discrimination methods are available (for example, Test Methods E1885, E2610, E2139, E3009, or E2164 as
examples of proper discrimination test execution; as previously stated, statistical hypothesis testing in an SGPE is not
recommended). To narrow down a sample set, which involves eliminating redundant sensory profiles and extreme outliers, sorting,
E3093 − 20
ranking, or scaling methods may be appropriate (6) as well as simple, open-ended questions (“write down all perceived flavors in
these samples”). If attribute scaling is needed, simple category scales are recommended, minimizing the number of categories.
Hedonic scales should not be used in an SGPE as a small group of employees would not be representative of the relevant consumer
group. For some objectives (such as describe or Discover Sensory Dimensions), collecting open-ended comments may be the best
approach.
7.12 Implement—Once the sensory method has been identified, the sensory professional should set the context for the
evaluation, deciding how much background and objective information is to be shared with participants. This includes a general
discussion of the issue at hand and identifying the sensory criteria that are relevant to the sensory question to be answered. While
too much or inappropriate information may bias the evaluation, the simple instruction to “taste these samples” or, in the case of
non-food items, “evaluate these samples” is generally not sufficient to focus participants’ attention and should be avoided. For
example, the sensory professional should tell the SGPE participants what type of products they will be tasting and, broadly, what
decision will be made. Further, instructions should indicate how many products will be evaluated, in what order, and on what the
SGPE participants should focus. The reveal of the product’s identity should be provided at the end of the independent evaluations
and after the discussion. If it is necessary to reveal the product identity before the start of the evaluations, then the sensory
professional and participants should understand that the outputs will be based on more than just the sensory experience. Post
evaluation, more information about the objective and further discussion of the r
...