ASTM E3313-24

Standard Guide for Communication of Assessor and Panel Performance

SIGNIFICANCE AND USE
4.1 The procedures recommended in this guide can be used by a panel leader to provide assessors and panels feedback: (1) on their data-based performance, (2) on any behavior changes that are needed to improve their performance, and (3) to motivate assessors to remain engaged with the panel tasks. The aim of all these types of feedback is to ensure the generation of repeatable and valid data.  
4.2 This guide provides direction for how to achieve mutually beneficial feedback exchanges between assessors and panel leaders.
SCOPE
1.1 This guide provides guidance to sensory panel leaders on how to deliver performance feedback to trained sensory assessors and panels. This guide is not intended to be used by individual assessors or anyone unfamiliar with the panel.  
1.2 This guide covers recommended feedback given throughout assessor training, panel development, and ongoing assessor and panel monitoring.  
1.3 This guide examines aspects of feedback including: types, when to provide, effective delivery, and alignment to performance expectations for assessors.  
1.4 Descriptive, discrimination, and quality panels are within the scope of this guide.  
1.5 This guide does not cover consumer panels (qualitative or quantitative).  
1.6 Units—The values stated in SI units are to be regarded as the standard. No other units of measurement are included in this standard.  
1.7 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.  
1.8 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

General Information

• Status: Published
• Publication Date: 14-Feb-2024
• ICS: 67.240 - Sensory analysis
• Technical Committee: E18 - Sensory Evaluation
• Drafting Committee: E18.05 - Method Applications - Specific Uses or Assessors

Relations

• Replaces: ASTM E3313-23a - Standard Guide for Communication of Assessor and Panel Performance
• Effective Date: 15-Feb-2024
• Refers: ASTM E456-13a(2022)e1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Apr-2022
• Refers: ASTM E456-13a(2022) - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Apr-2022

Overview

ASTM E3313-24: Standard Guide for Communication of Assessor and Panel Performance provides structured guidance for sensory panel leaders to deliver effective performance feedback to trained sensory assessors and panels. Developed by ASTM International, this guide is designed to empower panel leaders with communication tools and strategies that enhance assessor engagement, data reliability, and overall panel effectiveness throughout training, panel development, and ongoing monitoring.

Effective communication and feedback are fundamental for maintaining high-quality, repeatable, and valid sensory evaluation data, which underpin critical business and research decisions in product development and quality assurance.

Key Topics

• Purpose of Feedback: This standard emphasizes the importance of feedback for improving assessor skills, modifying unhelpful behaviors, and motivating ongoing participation. Timely, constructive feedback increases the repeatability and reliability of panel data.

• Types of Panels Covered: The guide applies to descriptive panels, discrimination panels, and quality assurance panels. It does not address consumer panels.

• Feedback Strategies: It outlines when and how to provide feedback during various stages:

  • Initial assessor training
  • Project or product-focused training
  • Post-evaluation data review
  • Regular panel maintenance

• Feedback Methods: Multiple feedback types are recommended, including:

  • Correct response feedback - Sharing the correct answer with assessors.
  • Confirmation feedback - Telling assessors whether their answers were correct or not.
  • Immediate feedback - Provided within a session for rapid adjustment.
  • Delayed feedback - Provided after analysis for in-depth discussion.
  • Behavioral feedback - Addressing group interactions, protocol adherence, and specific conduct.

• Criteria for Effective Feedback: Respect, honesty, clarity, and documentation are paramount. Feedback should align with performance expectations and be tailored to individual or group needs. It must also be actionable and accompanied by follow-up opportunities.

• Psychological Impacts: Properly delivered feedback increases assessor motivation and confidence while minimizing confusion and competition within panels.

Applications

ASTM E3313-24 is intended for use by sensory panel leaders in industries such as food & beverage, cosmetics, pharmaceuticals, and other product sectors where sensory evaluation is vital. Practical applications include:

• Training New Assessors: Establishes protocols for onboarding, skill-building, and performance calibration against panel standards.
• Panel Development: Supports focused training to address specific projects, product categories, or methodology changes.
• Panel Monitoring: Facilitates ongoing assessment and retraining efforts to uphold or improve data quality, manage panelist attrition, and ensure long-term panel effectiveness.
• Quality Assurance: Guides the implementation of feedback systems to maintain product standards and compliance with internal or regulatory requirements.

By following this guide, organizations can ensure that sensory panels produce high-quality, actionable data, directly informing product development, quality control, and innovation processes.

Related Standards

For comprehensive panel and assessor performance management, consider integrating ASTM E3313-24 with the following related ASTM standards:

• ASTM E253: Terminology Relating to Sensory Evaluation of Materials and Products
• ASTM E456: Terminology Relating to Quality and Statistics
• ASTM E3000: Guide for Measuring and Tracking Performance of Assessors on a Descriptive Sensory Panel

These standards collectively enhance the accuracy, consistency, and credibility of sensory evaluation data, supporting best practices in sensory panel leadership and assessor performance communication.

---

Keywords: ASTM E3313-24, sensory panel feedback, assessor performance, panel leader guidance, sensory evaluation, panel training, feedback strategies, product quality, data reliability, quality assurance panels, descriptive panels, discrimination panels.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E3313 − 24
Standard Guide for
Communication of Assessor and Panel Performance
This standard is issued under the fixed designation E3313; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope E253 Terminology Relating to Sensory Evaluation of Mate-
rials and Products
1.1 This guide provides guidance to sensory panel leaders
E456 Terminology Relating to Quality and Statistics
on how to deliver performance feedback to trained sensory
E3000 Guide for Measuring and Tracking Performance of
assessors and panels. This guide is not intended to be used by
Assessors on a Descriptive Sensory Panel
individual assessors or anyone unfamiliar with the panel.
1.2 This guide covers recommended feedback given
3. Terminology
throughout assessor training, panel development, and ongoing
3.1 Definitions—For definitions of terms relating to sensory
assessor and panel monitoring.
analysis, see Terminology E253. For terms relating to statistics,
1.3 This guide examines aspects of feedback including:
see Terminology E456.
types, when to provide, effective delivery, and alignment to
performance expectations for assessors. 4. Significance and Use
1.4 Descriptive, discrimination, and quality panels are
4.1 The procedures recommended in this guide can be used
within the scope of this guide. by a panel leader to provide assessors and panels feedback: (1)
on their data-based performance, (2) on any behavior changes
1.5 This guide does not cover consumer panels (qualitative
that are needed to improve their performance, and (3) to
or quantitative).
motivate assessors to remain engaged with the panel tasks. The
1.6 Units—The values stated in SI units are to be regarded
aim of all these types of feedback is to ensure the generation of
as the standard. No other units of measurement are included in
repeatable and valid data.
this standard.
4.2 This guide provides direction for how to achieve mutu-
1.7 This standard does not purport to address all of the
ally beneficial feedback exchanges between assessors and
safety concerns, if any, associated with its use. It is the
panel leaders.
responsibility of the user of this standard to establish appro-
priate safety, health, and environmental practices and deter-
5. Introduction to Feedback
mine the applicability of regulatory limitations prior to use.
5.1 What Is Feedback?—Feedback is the process of inform-
1.8 This international standard was developed in accor-
ing a trained assessor of their performance to maintain or
dance with internationally recognized principles on standard-
improve repeatability and validity of their responses. For a
ization established in the Decision on Principles for the
sensory assessor or panel, feedback can be the act of a panel
Development of International Standards, Guides and Recom-
leader sharing or informing assessors of their own results or
mendations issued by the World Trade Organization Technical
those of the panel or both; it can be providing positive or
Barriers to Trade (TBT) Committee.
negative reinforcement to change or correct behaviors; or it can
be acting as a “cheerleader” to instill confidence and increase
2. Referenced Documents
motivation.
2.1 ASTM Standards:
5.2 Reasons to Provide Feedback:
5.2.1 Performance—Providing appropriate and timely feed-
This guide is under the jurisdiction of ASTM Committee E18 on Sensory
back on an assessor’s or panel’s results contributes to stable
Evaluation and is the direct responsibility of Subcommittee E18.05 on Method
performance, performance improvement, and delivery of reli-
Applications - Specific Uses or Assessors.
able and robust results. It can help to make an assessor more
Current edition approved Feb. 15, 2024. Published February 2024. Originally
skilled at giving the same response for the same task under the
approved in 2022. Last previous edition approved in 2023 as E3313 – 23a. DOI:
10.1520/E3313-24.
same conditions.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
5.2.2 Adjusting Behaviors—Feedback is a tool that can be
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM
used to address panel/assessor issues if behaviors or results are
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website. not as expected. For example, if the assessor has scored an
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3313 − 24
attribute much lower than the panel, resulting in a different 5.4.3.1 Be respectful and honest. The panel leader needs to
sample rank order, then they will need feedback on how to tell the assessor(s) how they are doing, where they are
adjust their scoring. However, the panel leader should keep in fantastic, and where they need help;
mind that, depending on the method being used, and the 5.4.3.2 Assessors should be told that their feedback is a vital
sensitivity of the assessors, the outlier assessor can have useful part in the success of the panel; and
insights to contribute to the data. Feedback can also inform 5.4.3.3 When giving negative feedback related to perfor-
assessors of any corrective actions needed to address negative mance issues:
behaviors impacting the panel or highlight any positive impact (1) Determine from them where they think they need help
they may have on the panel. and devise a plan to fix the issues,
(2) Be open-minded to out-of-the-box solutions,
5.2.3 Motivation—Feedback should provide motivation for
(3) Give focused encouragement during the panel session if
the panels/assessors to continue learning and developing their
they make improvements in performance or behavior,
skills. It can help to increase panel/assessor credibility,
(4) Give deadlines for performance to meet expectations,
confidence, and satisfaction, which can then impact the
and
frequency, duration, and intensity of training.
(5) Use data and specific examples to support the feedback
5.3 Impact of Feedback:
given.
5.3.1 Feedback can have both positive and negative effects
5.4.4 Document—Ensure all feedback given and received is
on assessors. The choice for a panel leader to give feedback
documented to support development of the assessor and also
depends on the situation and the impact the feedback will have
for corrective action (for example, dismissal of assessor from a
on the recipient. When delivered correctly, feedback can
panel) if needed in the future.
deliver many positive consequences such as:
5.4.5 Evaluate—Use self-reflection to ensure your assessors
5.3.1.1 More accurate, relative, and consistent results;
understood the feedback and conduct a follow-up of discussed
topics in feedback through observation or other discussions.
5.3.1.2 Psychological benefits such as an increase in asses-
sor confidence and motivation; and Allow assessors to evaluate how feedback was given and
express what worked and did not work as feedback was given.
5.3.1.3 Reducing confusion, conflicts, and competition
among assessors.
5.5 Setting Performance Expectations for Assessors:
5.3.2 Negative consequences are also possible if feedback is
5.5.1 It is important to make assessors aware of the level of
given incorrectly, including: performance that is expected for the stage of development they
are at and the role they are undertaking. Performance expec-
5.3.2.1 Negative impact on results (for example, increased
tations can vary depending on if the assessor is new or
data variability);
experienced. The consequences of poor performance, for
5.3.2.2 Psychological effects that may create a negative
example, extra training sessions or even dismissal, need to be
environment for assessors (for example, impact on group
explained to assessors at the beginning of their training. For
dynamics, decrease in individual motivation) and increase
employee panels, it is important to make attendance expecta-
panel turnover; and
tions clear, how panel participation relates to their current role,
5.3.2.3 Confusion for the assessor if too much, or
and what level of performance they will need to maintain to
inconsistent, feedback is given.
remain on the panel.
5.4 How to Give Feedback:
5.5.2 To help assessors understand why their performance is
monitored, the panel leader needs to clearly state the impor-
5.4.1 Initiate—The panel leader needs to think about how to
tance of quality data and the critical role the panel results play
initiate feedback with an assessor or panel. Feedback can be
in business decisions.
spontaneous, on a scheduled basis, one-on-one, or in a group
depending on the type of feedback being given (see 5.8).
5.6 When to Give Feedback—Assessors should be told that
5.4.2 Plan and Formulate—If planning feedback in
feedback can occur during training and monitoring activities.
advance, the panel leader needs to think of the reasons for
There are generally four occasions in which trained assessors
giving the feedback and outline the steps that need to be taken
require feedback: (1) training of new assessors, (2) project or
to deliver it. If the panel leader is giving individual feedback to
product category training, (3) post-product assessment and data
an assessor, especially if giving spontaneous feedback, they
collection, and (4) panel monitoring and maintenance.
need to consider their personalities, learning styles, or needs.
5.6.1 Training of New Assessors:
Effective feedback involves dialog with assessors in a way that
5.6.1.1 Newly recruited assessors with no previous experi-
acknowledges good performance or identifies areas of im-
ence will be learning many new skills and need slightly
provement.
different feedback from that given to more experienced asses-
5.4.3 Exchange—Everyone has their own personality and sors. Regardless, if new assessors are trained in separate
will respond to feedback differently. It is important for the sessions with other new assessors before being added to a
panel leader to adjust their communication style to fit the panel, or if they are added directly to an existing panel, they
emotional needs of individual assessors at the time of the will need to be given information about the sensory evaluation
feedback. To have a successful feedback exchange with asses- methods they will be using, how they are progressing in
sors or a panel, the panel leader needs to consider the following developing their skills, and how well they are calibrating with
points: other assessors. Assessors new to the panel should be made
E3313 − 24
aware that it is expected that they will make mistakes and be needed to improve data quality. It can also be a form of
initially have a lower level of performance compared to more motivation, for example, having a senior manager thank the
experienced assessors. panel for their contribution to a project.
5.6.1.2 A training session should be given to assessors
5.8 Types of Feedback:
showing the types of performance diagnostics that will be
5.8.1 There are five main types of feedback that can be used
given to them. Various tables and graphics should be prepared
with assessors and panels. Each type of feedback may be used
and explained to the panel. Assessors should also be told that
for different tasks during the different phases of panel devel-
performance feedback may also include behavioral aspects
opment and each has advantages and disadvantages.
such as how assessors interact with the group during discussion
5.8.1.1 Correct Response—Informs the assessor of the cor-
and how disagreements are handled.
rect answer.
5.6.2 Project or Product Category Training—In this training
5.8.1.2 Confirmation Feedback—Informs the assessor
phase, the assessors will be gaining specific knowledge about
whether a response was correct or incorrect. The correct
products being evaluated in a specific project or within a
response is not given to them.
product category. They will be expected to learn about specific
5.8.1.3 Immediate Feedback
(within a session)—
attributes and references and learn how to calibrate with the
Information provided after the assessor gives their response; it
panel as a whole. The final performance level of the assessors
is most beneficial when received during training. Immediate
and panel will be used to validate that the panel is ready to
feedback can be given directly after the assessor gives a
move to the product evaluation and data collection phase of a
response (refer to Case Study 4, Appendix X4) or it can be
project.
given very soon after the assessor’s response is entered during
5.6.3 Post-Product Assessment and Data Collection—The
the same panel session. Immediate feedback can be used
type of sensory methodology (for example, discrimination
during training, but is not recommended during data collection
versus descriptive tasks versus quality evaluations) used may
as this can bias the test results.
dictate whether or not to give feedback after a product
5.8.1.4 Delayed Feedback—Often, it is not possible to give
assessment or when a project is completed. Discrimination
feedback immediately. The data may require analyzing, plots
tests may not always require feedback and it can have a
prepared, and feedback planned. Be aware that learning may be
negative impact if the assessor consistently gets the test wrong.
reduced, especially with longer gaps between the activity and
A descriptive panel will most likely benefit from feedback to
the feedback.
help facilitate learning and skill improvement. The feedback
5.8.2 Tables 1-3 summarize which type of performance
given during this phase is more specific and relates to the
feedback you can use during each type of panel and phase of
assessor/panel performance results. It helps to continue the
panel development and also provide some examples.
development of the assessors’ skills and performance level.
6. Feedback During Descriptive Panel Training
5.6.4 Panel Monitoring and Maintenance—General feed-
back on panel performance is good for monitoring assessor
6.1 Descriptive panels profile products by quantifying attri-
performance over time and maintaining the panel’s proficiency
bute intensities on rating scales. They are usually trained in
level between projects. Maintenance activities may include
specific profiling methods and familiarized with specific prod-
conducting specifically designed tests to evaluate panel/
ucts or product categories. These panels should demonstrate
assessor performance; retraining on existing methods and
good repeatability of results. Attrition is an issue for longstand-
products as deemed necessary; expanding/enhancing skills
ing panels, requiring integration of new members on occasion.
required for new methods, attributes, and products; dealing
To maintain high-performing panels, training is an ongoing
with panel attrition; and so forth.
effort as is performance monitoring (with feedback).
5.7 Individual Assessor Versus Group Feedback to the
6.2 Feedback During Training of New Assessors:
Panel as a Whole:
6.2.1 For brand new assessors without previous descriptive
5.7.1 Feedback is important for both individual assessors,
analysis experience, it is critical for them to understand that
who perform the task, and the panel as a whole, who produce
many of the skills needed to become a trained assessor on a
the results used for decision making.
descriptive panel are foreign to normal product usage patterns,
5.7.2 Individual feedback is used as a status check for the but with training, they will learn to grasp the concepts and
individual assessor and provides them with information about
apply this new knowledge to the evaluation process. They
their own performance versus that of the panel. It may also be
should be encouraged to ask questions at each session and
needed for difficult situations, for example, when the panel
understand that they will make mistakes during this phase of
leader needs to draw attention to or correct an assessor’s
development. In general, training should commence with
behavior. It can also be used to motivate an assessor if they
simple tasks then proceed to more complicated work as the
have improved their performance or made a positive contribu-
assessors learn. Continuous feedback on each new task is
tion to the panel.
important to the learning process. Assessors should be made
5.7.3 Group feedback is given to the panel as a group. It can
be used to highlight data performance issues, for example,
Findlay, C. J., Castura, J. C., and Lesschaeve, I., “Feedback calibration: A
presenting plots after a descriptive analysis/profiling project
training method for descriptive panels,” Food Quality and Preference, Vol 18, No.
and discussing the attributes that have a lot of disagreement
2, 2007, pp. 321-328, ISSN 0950-3293, https://doi.org/10.1016/
between assessors and what training or behavior changes will j.foodqual.2006.02.007.
E3313 − 24
TABLE 1 Types of Feedback Given during Different Phases of a Descriptive Panel
Phase Types of Appropriate Feedback Examples
Training of new assessors Confirmation response Telling assessors that they had the incorrect
response for a four-sample sweet ranking test.
Correct response Giving assessors the correct order of samples in a
four-sample sweet ranking test.
Immediate feedback (within a session) Telling the assessor who got the sweet ranking test
to go back and taste the samples in the sweet
ranking test and to focus on how they ranked the
less intense samples.
Praising an assessor for accomplishing a new task
(motivation).
Requesting more explanation from an assessor
when they are struggling with a new attribute or
task.
Asking assessors to talk one at a time during panel
discussion (behavior change).
Delayed feedback Discussing any changes in scoring behaviors that
might be needed after reviewing a plot of mean
scores and standard deviation for each assessor
versus the panel.
Giving positive feedback on performance
improvement for a difficult attribute when viewing the
plot.
Project or product category training Immediate feedback (within a session) Confirming that an assessor has scored the intensity
of an attribute close to the desired target or to the
panel average.
Asking assessor to adjust their scores if using the
consensus method.
Requesting an assessor to review and adjust their
attribute scores during a group training session.
Delayed feedback Discussing any changes in scoring behaviors that
might be needed after reviewing a plot of mean
scores and standard deviation for each assessor
versus the panel.
Post-product assessment and data collection Delayed feedback Praising good performance of specific assessors
versus the panel when showing a plot of attribute
mean scores and standard deviations to the panel.
Highlighting attributes with poor performance that
will need more training.
Panel monitoring and maintenance Delayed feedback Informing individual assessors of the attributes they
are not performing well on during the one-year
review of results from a two-year storage trial.
aware of the “correct” response, where applicable/relevant, and can be used to determine what feedback to give and supple-
should re-experience test sample(s) so that they can align their ment feedback. The following formats are commonly used:
perception to the expected response. The “correct” response 6.3.2.1 Attribute discrimination (good versus poor discrimi-
can be determined by past panel work within the company,
nation attribute lists),
literature, or the current panel once it is more experienced.
6.3.2.2 Graphical illustrations (including boxplots, mean
6.2.2 Immediate feedback during initial training can help to
and target ranges, interaction plots, and so forth),
ensure poor habits are corrected early. Examples of poor habits
6.3.2.3 Reports or presentations, or both (often providing a
include not following protocol(s); not allowing adequate time
historical overview and depiction of performance over time),
between samples; adopting misinterpretation or misuse of scale
6.3.2.4 Targets/references (panel leader provides qualitative
methods; habitual, non-discriminating use of scales; and mis-
or quantitative examples, or both),
understanding and incorrect rating of specific attributes.
6.3.2.5 Correct answer or agreement, and
6.3 Feedback During Project Training: 6.3.2.6 Individual assessor rank compared to panel as a
whole.
6.3.1 During project training, two-way communication be-
tween the panel leader and assessors should be open and 6.3.3 Feedback about assessor performance can occur after
frequently encouraged. The panel may debate and discuss as collecting individual raw scores from each assessor, either on
they try to arrive at common understanding and potentially paper or electronically. The panel leader can provide means,
resolve differences for new sample sets or when learning new ranges, and other statistical analyses, including graphs, for the
methods or attributes. discussion with the panel. This process will allow for the
6.3.2 During training sessions, assessors commonly assess assessors to see how they perform individually and, in
samples and references during group discussions. Various tools comparison, with the rest of the panel.
E3313 − 24
TABLE 2 Types of Feedback Given During Different Phases of a TABLE 3 Types of Feedback Given during Different Phases of a
Discrimination Panel Quality Panel
Phase Discrimination Examples Phase Quality Examples
Training of new Correct feedback Assessor is told they Training of new Correct feedback Assessor is told they
assessors answered a tetrad test assessors correctly identified the
incorrectly and shown sample spiked with
the correct groupings. butyric acid.
Confirmation feedback Assessor is told they Confirmation feedback Telling assessors that
did not get a triangle they had the incorrect
test correct and is not response for a four-
told what the correct sample ranking test.
answer was. Immediate feedback Praising an assessor
Immediate feedback Assessor is told they for accomplishing a
(within a session) should reassess the new task (motivation).
triangle test samples
and focus on the bitter Project or product Correct feedback Assessor is told they
taste in the samples. category training answered a triangle test
incorrectly and shown
Project or product Correct feedback Assessor is told they the correct groupings.
category training answered a triangle test Immediate feedback Panel leader praises an
incorrectly and shown assessor for correctly
the correct groupings. identifying and
Immediate feedback Panel leader praises an recognizing the spiked
assessor for getting samples in a training
three triangle tests session.
correct in the training
session. Post-product Delayed feedback Project leader shares
assessment and data business outcome from
Post-product Correct feedback Assessor is told they collection the panel evaluation.
assessment and data answered the tetrad
collection test correctly.
Panel monitoring and Immediate feedback After a “true-to-type”
maintenance monitoring test,
Panel monitoring and Delayed feedback Assessors are shown a assessors are told the
maintenance count of how many correct descriptions and
triangle tests they have allowed to reassess the
participated in over the samples.
past month and given a
Delayed feedback Panel is informed of the
gift voucher if they attributes they need
attended all ten tests. further training on after
reviewing results of
daily evaluations (see
case study for more
information).
6.3.4 Based on the mean and the range of scores obtained
for each attribute, the panel can discuss why they agree or
disagree with the average rating. The panel should re-
experience the product, assessors should determine if they
would like to adjust their scores, and the panel should come to
tions for performance should be described. Of particular
agreement about whether the mean accurately reflects the
interest to panel members are the following questions: (1) how
product attribute under discussion. Continuous feedback from
did the panel do overall, both positive and constructive
the panel leader is important to facilitating this cyclic process.
examples (2) how can the panel improve, (3) how did the
6.3.5 Individual Assessor Feedback—Assessors should un-
individual assessor do overall, both positive and constructive
derstand the importance of the data they give to the project and
examples, and (4) where can individual assessor improvements
how that influences decisions. The panel leader should con-
be made.
sider the assessors’ background knowledge when sharing
6.4 Feedback After Data Collection—The same tools and
performance tables or graphs. Make sure they understand the
type of feedback given to assessors and the panel during
terms (for example, mean and standard deviation) and how to
training (6.3) can also be used after data collection is complete.
read the graphs used to represent data. Panel leaders need to
Giving this type of feedback is dependent on any effect that it
explain what the assessor should get from the information and
may have on the project objectives and final results. For
how they can use it to improve their performance. Examples of
example, information shared after a one-off project may be
good and poor quality performances should be pointed out and
different from that giving during an ongoing study such as a
discussions with the assessor can identify and resolve issues in
shelf-life study.
evaluating attributes of concern.
6.3.6 Panel Feedback—Panel performance results can be 6.5 Feedback During Panel Monitoring—Panel mainte-
shared to both inform and motivate panels, though make sure nance involves preserving the performance level of the panel
to consider the background of the panel and project to for continued use. Training exercises are designed to ensure the
determine if sharing such results is appropriate. An overview of panel does not drift and that they maintain their ability to
satisfactory and unsatisfactory performances should precede discriminate at the desired level of sensitivity needed to make
sharing of specific results so that members of the panel can business decisions. Activities may include conducting specifi-
focus their attention where needed. Also, the level of expecta- cally designed tests to evaluate panel/assessor performance;
E3313 − 24
retraining on existing methods and products; and expanding/ possible, to confirm or correct their result. Replication of the
enhancing skills required for new methods, attributes, and training test(s) on different occasions, again with immediate
products. feedback, can improve the assessor’s ability in that discrimi-
6.5.1 Individual Assessor Feedback: nation method. For example, they may learn that they work
6.5.1.1 Similar feedback can be given as described in 6.3.5. better in a particular discrimination test if they make their
If an assessor’s performance deteriorates over time, it will be decision based on their initial judgement or, alternatively, that
necessary to give them repeated feedback on the areas in which they need to assess the samples a couple of times to confirm
they need to improve and how to improve. Make sure to their answer.
consider the information in 5.7 when having this type of
7.4 Feedback During Project Training:
discussion.
7.4.1 Immediate feedback to the assessor regarding their
6.5.1.2 If performance does not improve, it may eventually
“correct response” is only recommended for training tests and
become necessary to remove the assessor from the panel. If
not real test situations. There are several reasons for this. If the
applicable, consult and involve Human Resources when plan-
sensory facility is mainly conducting similarity tests and an
ning to have this type of discussion with an assessor.
assessor is only able to identify the odd sample correctly in a
6.5.2 Panel Feedback—Similar feedback can be given to
series of triangle tests around 33 % of the time, being given
the panel as described in 6.3.6.
constant feedback that they were “incorrect” may lower
motivation to be on the panel. If a sensory facility replicates all
7. Feedback During Discrimination Testing
discrimination tests, immediate feedback after each replication
7.1 In a discrimination test, the assessor’s task is to deter-
can influence the results of the next replicate. Also, when
mine if the products that they are presented with are different
working on a large project such as salt or sugar reduction,
from each other. There are many different discrimination tests
giving assessors immediate feedback about whether they were
available depending on the test objectives, product types, and
“correct” or not may well impact their performance in the next
resources available, but the feedback approach for screened
related tests by giving them clues as to the objective of the
and trained assessors is quite similar for all methods.
tests.
7.2 Discrimination panels are typically company employees
7.4.2 However, delayed feedback relating to project prog-
or trained panels; these are assessors who continuously partici-
ress (without the specific detail about the project) can be very
pate on this type of panel. Feedback plays an important role in
motivating to the assessors, as well as a simple “thank you for
motivating them to return for subsequent tests. The feedback
good attendance.”
for overall or unspecified discrimination tests (for example,
7.5 Feedback During Panel Monitoring—Discrimination
same-different or triangle) differs slightly from that for
assessors who test on a regular basis should receive feedback
attribute-specific discrimination tests (for example, directional
on their performance both as individuals and a group.
paired comparison or two alternative forced choice) as de-
7.5.1 Individual Assessor Feedback:
scribed in 7.3.
7.5.1.1 Monitoring can be simply performed by recording
7.3 Feedback During Training of New Assessors:
each assessor’s results over time and comparing to the panel’s
7.3.1 When training new assessors in any discrimination
result: in fact, some software systems will offer this as
task, the main feedback objectives are to ensure that the
standard. If the panel found a statistically significant difference
assessor knows how to take part in the test and instill
between the products, each individual assessor’s result is
confidence in the assessors for subsequent real test situations.
compared to this. The sensory science function can then set
The products used for training are generally known to be
their own action standard for the performance monitoring such
different and the objective is to qualify the assessor for future
as: “if an assessor does not identify a difference in more than
testing situations for that discrimination method, for example,
40 % of tests when the panel as a whole has found a difference,
duo-trio or tetrad.
validation tests (see Note 1) will be performed to check that the
7.3.2 In attribute-specific discrimination tests, a further
assessor still meets the requirements for the discrimination
objective is added to the training: ensuring that the assessor
testing panel.”
understands and recognizes the attribute of interest. For
NOTE 1—A validation test is a test in which the result is already known
example, if the two-alternative forced choice training test is to
and an assessor’s performance can be compared directly to this.
determine which of two beers is more bitter, the assessor shall
7.5.1.2 Recording the results of all tests will also help
be able to understand that they are looking for a difference in
identify those assessors who are able to detect consistently
bitterness and not, say, carbonation or sourness, but also that
small differences or are particularly adept at taint detection, for
they can detect and quantify the levels of bitterness in the beers
example. However, this information is rarely used for feedback
to be able to recognize the difference in the samples presented.
except to report and recognize attendance.
Training and feedback for attribute-specific discrimination
tests, therefore, also include tests to help the assessor recognize 7.5.1.3 When validation tests are performed regularly, in-
and quantify the attribute of interest by gradually decreasing forming the assessors if they were “correct” in their sample
the level of difference in the attribute. choice can be very motivating for the assessors; however, be
7.3.3 When training new assessors in discrimination tests, it careful that the differences between the products do not
can be very helpful to give immediate feedback to the assessor become common knowledge or further validation study prod-
on their performance so that they may reassess the products, if ucts will need to be sourced.
E3313 − 24
7.5.2 Panel Feedback—Group feedback for a discrimina- 8.5 To help assessors understand desirable product
tion panel usually consists of a summary of the number of tests attributes, product identification training and feedback can be
conducted and a thank you from the panel leader or project given. In training, assessors are presented with a sample whose
leader for the assessors’ contributions. If the panel takes part in brand is not given and they shall correctly identify the brand.
any comparative testing with panels in other locations, sharing Assessor performance on brand identification should be
the overall outcome of these tests, for example, that the panel tracked and compared against preset validation parameters (for
managed to identify the difference as a team, can also be example, 75 % identification of each brand).
interesting and aspirational for the assessors. After testing is
8.6 Assessors should also be trained and provided feedback
complete, assessors may be motivated by learning the objective
on specific off attributes that can occur in products they
of the test and how their work saved the company money,
evaluate. Typically, this is done by adding a specific off
accomplished a specific objective, helped to launch a new
attribute to the product and presenting it to the assessor either
product, and so forth. However, be careful that the information
blind during normal assessment or in a training setting in which
shared will not bias the assessors for future projects.
they are aware doctored samples are present. Both detection,
which is the ability of the assessor to perceive an atypical
8. Sensory Quality Testing
aspect, and recognition, which is the ability of the assessor to
8.1 Different types of sensory quality panels are used in
identify correctly the off note, should be tracked and compared
industry including: quality assurance panels that support as-
against preset validation parameters as with brand identifica-
sessment of incoming ingredients and outgoing plant produc-
tion.
tion; shelf-life panels to monitor quality of the product over
8.7 Feedback on assessor repeatability is also important.
time and under different storage conditions; panels dedicated to
Some methods may already incorporate this type of data
new product development initiatives, sourcing, and procure-
analysis, but others such as in/out should have repeated
ment; and other applications in which product quality is a
samples included and are analyzed with a qualitative approach.
concern.
If an assessor is consistent in saying a sample is in or out of
8.2 Sensory quality panels are typically comprised of
specification and their comments are the same between the two
dedicated assessors very familiar with the products being
samples, they would be considered repeatable.
tested. The assessors can be internal employees of the company
8.8 Individual assessor responses should also be examined
or external assessors. Usual/routine evaluations often involve
for agreement with the broader panel response. Though certain
comparison to controls or retains. As opposed to descriptive
methods such as difference from control and descriptive
panels that involve detailed profiling, the quality assurance
analysis easily allow for such analyses, qualitative methods
(QA) sensory quality panels generally are tasked to rate overall
may be used for agreement for methods such as in/out. For
match to a target or control in a very efficient, rapid manner to
example, it is straightforward to examine if an assessor
keep pace with production and ensure product sensory speci-
indicates samples are out of specification more often than the
fications are satisfied. As a result, quality panels often test a
panel does on average. If assessors make comments on
high volume of products on a daily basis and generally work
samples, these too can be compared qualitatively between a
more hours in a day assessing the sensory aspects of products.
single assessor and the panel, and discrepancies can be given as
Performance expectations should be clearly communicated to
feedback to individual assessors.
the assessors when they start in their role as this makes
marginal or poor performance more easily addressed.
8.9 For all types of quality performance, it may be useful to
8.3 Typical quality methods may be as simple as discrimi- compare assessor and panel performance in brand
identification, attribute recognition, repeatability, and agree-
nation tests (for example, duo-trio or tetrad) and designating a
sample as in or out of specification (“In/Out” or A-not-A) or ment across multiple production facilities to ensure the same
assessor performance, product(s) characteristics, and quality
more diagnostic such as the relative-to-reference (directional-
degree-of-difference) scale, simple descriptive analysis, or levels are maintained at all locations.
more complex shelf-life determination or confirmation. Train-
8.10 Feedback During New Assessor Training—
ing and monitoring performance with feedback involves en-
Communication regarding the method, product characteristics,
suring assessors are familiar with the control product(s) and the
ranges, and so forth during the training of a sensory quality
acceptable range of deviation, although some quality methods
panel is critical regardless of the type of panel (plant,
do not require the assessment of controls within the test itself
corporate, or external). Assessors will require clear instruction
or do not have the ability to keep controls stable. In these
on evaluation methodology and familiarization with the ingre-
instances, assessors refer to a written sensory specification.
dients and finished products that are being evaluated, as well as
8.4 Training and monitoring should also ensure that asses- the “on” and “off” notes. During the training, the assessors
sors are capable of determining that the intended attributes or should also develop an understanding of their own abilities, for
“on notes” are within the target intensity range and they are example, when they are anosmic to certain ingredient or
also capable of identifying any potential defects of “off notes” product characteristics or they are especially sensitive to other
of each product. Therefore, where possible, it is good practice aspects such as taints or “off-notes.” The training should allow
to present products containing both “on” and “off” notes to the assessors to understand that everyone has different sensitivities
quality assessors during training so that they can experience the and the role is as a group, all working together. This helps
full range of characteristics. boost confidence and morale of the whole quality panel. As
E3313 − 24
well as these physiological limitations, training should also 8.11.3.4 If the quality project involves descriptive analysis
include information about psychological errors; biases such as or descriptive analysis-type scaling approaches, please see
expectation, logical, and habitual errors are especially impor-
Section 6 for more information about feedback for these
tant for all quality assessors to understand so that they can help
methods. For discrimination tests (for example, difference from
prevent issues.
control and ranking) see Section 7. For specific quality
methods such as “in-out of specification,” there are two main
8.11 Feedback During an Ongoing Operation:
types of feedback: the results of an individual assessor can be
8.11.1 Feedback is essential for sensory quality panels, but
compared to the rest of the panel for a specific test or an
because of innate differences between a plant, corporate, or
individual’s results can be monitored over time. For the latter,
external quality panel, the type and delivery of feedback will
please see 8.12.
vary. If working on a project basis (for example, shelf life),
feedback on performance can be given during training on the
8.11.4 Panel Feedback—The same approach mentioned in
samples or category. If the panel is working in a continuous
8.11.3 can be used for panel feedback. Feedback can be given
production environment, feedback might be best given imme-
to a panel at the end of a discrete project or in a production
diately to individual assessors, or results may be collected and
environment regular feedback on performance over time, and
given as a summary report at a regular time point (for example,
product batches should be scheduled in as part of morning
daily or weekly).
panels or weekly/monthly trainings.
8.11.2 The sensory scientist should ensure that feedback
8.12 Feedback During Panel Monitoring and Maintenance:
does not have a detrimental impact on future testing programs.
For example, if the method of choice is the “in-out” method,
8.12.1 Quality panels typically become very familiar with
the assessors are in control of the decision-making process: if
the incoming ingredients and outgoing finished goods from
the success of their results are linked to an annual bonus, this
production and generate substantial amounts of data that can be
may lead to approval of all products. Compare this to the
tracked over time, for example, with statistical process control
potentially less biasing descriptive specification method in
(SPC). This method allows the sensory scientist to monitor the
which the sensory scientist makes the decision whether a
assessors as well as detect production issues before a product
product is “in” or “out” of specification based on the assessors’
goes out of specification. Once established, assessor and panel
results. For discrimination testing, often the “right” result is not
performance should be monitored on an ongoing regular basis
being able to identify correctly the difference, especially for
and feedback about performance levels given on a regular
minor ingredient or production changes, and assessors should
basis.
not be penalized for their answers. Assessors should be
8.12.2 Monitoring tests may include “out” samples (such as
rewarded for taking part, not just their results.
held products from previous production issues or spiked
8.11.3 Individual Assessor Feedback:
samples, for example, samples with an added taint or a specific
8.11.3.1 Assessors should be given feedback on their
attribute at a higher level), or replicates of specific samples.
attendance, success, and motivation to take part in the routine
Feedback should be given to each assessor directly after these
assessments. A simple “thank you” for attendance can be
tests for two main reasons: (1) to inform assessors that these
highly motivating and motivated assessors generally produce
validations are conducted as part of the routine program, (2) to
better data.
allow assessors to reassess the products to confirm or refamil-
8.11.3.2 Immediate feedback about an assessor’s perfor-
iarize themselves with the products in the test. This is espe-
mance can be given after a test has been completed or if data
are continuously being collected. Feedback can also be given at cially vital for “off” notes so that the assessor and the sensory
a regular time point (for example, daily, weekly, or monthly) scientist can build up their understanding of each assessor’s
via graphs or reports. Feedback on how an assessor is perform- abilities. The feedback should clearly state whether the asses-
ing over production time and batches is
...

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E3313 − 23a E3313 − 24
Standard Guide for
Communication of Assessor and Panel Performance
This standard is issued under the fixed designation E3313; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope
1.1 This guide provides guidance to sensory panel leaders on how to deliver performance feedback to trained sensory assessors
and panels. This guide is not intended to be used by individual assessors or anyone unfamiliar with the panel.
1.2 This guide covers recommended feedback given throughout assessor training, panel development, and ongoing assessor and
panel monitoring.
1.3 This guide examines aspects of feedback including: types, when to provide, effective delivery, and alignment to performance
expectations for assessors.
1.4 Descriptive, discrimination, and quality panels are within the scope of this guide.
1.5 This guide does not cover consumer panels (qualitative or quantitative).
1.6 Units—The values stated in SI units are to be regarded as the standard. No other units of measurement are included in this
standard.
1.7 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.8 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E253 Terminology Relating to Sensory Evaluation of Materials and Products
E456 Terminology Relating to Quality and Statistics
E3000 Guide for Measuring and Tracking Performance of Assessors on a Descriptive Sensory Panel
This guide is under the jurisdiction of ASTM Committee E18 on Sensory Evaluation and is the direct responsibility of Subcommittee E18.05 on Sensory
Applications--GeneralMethod Applications - Specific Uses or Assessors.
Current edition approved Oct. 1, 2023Feb. 15, 2024. Published October 2023February 2024. Originally approved in 2022. Last previous edition approved in 2023 as
E3313 – 23.E3313 – 23a. DOI: 10.1520/E3313-23A.10.1520/E3313-24.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3313 − 24
3. Terminology
3.1 Definitions—For definitions of terms relating to sensory analysis, see Terminology E253. For terms relating to statistics, see
Terminology E456.
4. Significance and Use
4.1 The procedures recommended in this guide can be used by a panel leader to provide assessors and panels feedback: (1) on
their data-based performance, (2) on any behavior changes that are needed to improve their performance, and (3) to motivate
assessors to remain engaged with the panel tasks. The aim of all these types of feedback is to ensure the generation of repeatable
and valid data.
4.2 This guide provides direction for how to achieve mutually beneficial feedback exchanges between assessors and panel leaders.
5. Introduction to Feedback
5.1 What Is Feedback?—Feedback is the process of informing a trained assessor of their performance to maintain or improve
repeatability and validity of their responses. For a sensory assessor or panel, feedback can be the act of a panel leader sharing or
informing assessors of their own results or those of the panel or both; it can be providing positive or negative reinforcement to
change or correct behaviors; or it can be acting as a “cheerleader” to instill confidence and increase motivation.
5.2 Reasons to Provide Feedback:
5.2.1 Performance—Providing appropriate and timely feedback on an assessor’s or panel’s results contributes to stable
performance, performance improvement, and delivery of reliable and robust results. It can help to make an assessor more skilled
at giving the same response for the same task under the same conditions.
5.2.2 CorrectingAdjusting Behaviors—Feedback is a tool that can be used to address panel/assessor issues if behaviors or results
are not as expected. For example, if the assessor has scored an attribute much lower than the panel, resulting in a different sample
rank order, then they will need feedback on how to correct and adjust their scoring. However, the panel leader should keep in mind
that, depending on the method being used, and the sensitivity of the assessors, the outlier assessor can have useful insights to
contribute to the data. Feedback can also inform assessors of any corrective actions needed to address negative behaviors impacting
the panel or highlight any positive impact they may have on the panel.
5.2.3 Motivation—Feedback should provide motivation for the panels/assessors to continue learning and developing their skills.
It can help to increase panel/assessor credibility, confidence, and satisfaction, which can then impact the frequency, duration, and
intensity of training.
5.3 Impact of Feedback:
5.3.1 Feedback can have both positive and negative effects on assessors. The choice for a panel leader to give feedback depends
on the situation and the impact the feedback will have on the recipient. When delivered correctly, feedback can deliver many
positive consequences such as:
5.3.1.1 More accurate, relative, and consistent results;
5.3.1.2 Psychological benefits such as an increase in assessor confidence and motivation; and
5.3.1.3 Reducing confusion, conflicts, and competition among assessors.
5.3.2 Negative consequences are also possible if feedback is given incorrectly, including:
5.3.2.1 Negative impact on results (for example, increased data variability);
5.3.2.2 Psychological effects that may create a negative environment for assessors (for example, impact on group dynamics,
decrease in individual motivation) and increase panel turnover; and
E3313 − 24
5.3.2.3 Confusion for the assessor if too much, or inconsistent, feedback is given.
5.4 How to Give Feedback:
5.4.1 Initiate—The panel leader needs to think about how to initiate feedback with an assessor or panel. Feedback can be
spontaneous, on a scheduled basis, one-on-one, or in a group depending on the type of feedback being given (see 5.8).
5.4.2 Plan and Formulate—If planning feedback in advance, the panel leader needs to think of the reasons for giving the feedback
and outline the steps that need to be taken to deliver it. If the panel leader is giving individual feedback to an assessor, especially
if giving spontaneous feedback, they need to consider their personalities, learning styles, or needs. Effective feedback involves
dialog with assessors in a way that acknowledges good performance or identifies areas of improvement.
5.4.3 Exchange—Everyone has their own personality and will respond to feedback differently. It is important for the panel leader
to adjust their communication style to fit the emotional needs of individual assessors at the time of the feedback. To have a
successful feedback exchange with assessors or a panel, the panel leader needs to consider the following points:
5.4.3.1 Be respectful and honest. The panel leader needs to tell the assessor(s) how they are doing, where they are fantastic, and
where they need help;
5.4.3.2 Assessors should be told that their feedback is a vital part in the success of the panel; and
5.4.3.3 When giving negative feedback related to performance issues:
(1) Determine from them where they think they need help and devise a plan to fix the issues,
(2) Be open-minded to out-of-the-box solutions,
(3) Give focused encouragement during the panel session if they make improvements in performance or behavior,
(4) Give deadlines for performance to meet expectations, and
(5) Use data and specific examples to support the feedback given.
5.4.4 Document—Ensure all feedback given and received is documented to support development of the assessor and also for
corrective action (for example, dismissal of assessor from a panel) if needed in the future.
5.4.5 Evaluate—Use self-reflection to ensure your assessors understood the feedback and conduct a follow-up of discussed topics
in feedback through observation or other discussions. Allow assessors to evaluate how feedback was given and express what
worked and did not work as feedback was given.
5.5 Setting Performance Expectations for Assessors:
5.5.1 It is important to make assessors aware of the level of performance that is expected for the stage of development they are
at and the role they are undertaking. Performance expectations can vary depending on if the assessor is new or experienced. The
consequences of poor performance, for example, extra training sessions or even dismissal, need to be explained to assessors at the
beginning of their training. For employee panels, it is important to make attendance expectations clear, how panel participation
relates to their current role, and what level of performance they will need to maintain to remain on the panel.
5.5.2 To help assessors understand why their performance is monitored, the panel leader needs to clearly state the importance of
quality data and the critical role the panel results play in business decisions.
5.6 When to Give Feedback—Assessors should be told that feedback can occur during training and monitoring activities. There
are generally four occasions in which trained assessors require feedback: (1) training of new assessors, (2) project or product
category training, (3) post-product assessment and data collection, and (4) panel monitoring and maintenance.
5.6.1 Training of New Assessors:
5.6.1.1 Newly recruited assessors with no previous experience will be learning many new skills and need slightly different
feedback from that given to more experienced assessors. Regardless, if new assessors are trained in separate sessions with other
new assessors before being added to a panel, or if they are added directly to an existing panel, they will need to be given
information about the sensory evaluation methods they will be using, how they are progressing in developing their skills, and how
E3313 − 24
well they are calibrating with other assessors. Assessors new to the panel should be made aware that it is expected that they will
make mistakes and initially have a lower level of performance compared to more experienced assessors.
5.6.1.2 A training session should be given to assessors showing the types of performance diagnostics that will be given to them.
Various tables and graphics should be prepared and explained to the panel. Assessors should also be told that performance feedback
may also include behavioral aspects such as how assessors interact with the group during discussion and how disagreements are
handled.
5.6.2 Project or Product Category Training—In this training phase, the assessors will be gaining specific knowledge about
products being evaluated in a specific project or within a product category. They will be expected to learn about specific attributes
and references and learn how to calibrate with the panel as a whole. The final performance level of the assessors and panel will
be used to validate that the panel is ready to move to the product evaluation and data collection phase of a project.
5.6.3 Post-Product Assessment and Data Collection—The type of sensory methodology (for example, discrimination versus
descriptive tasks versus quality evaluations) used may dictate whether or not to give feedback after a product assessment or when
a project is completed. Discrimination tests may not always require feedback and it can have a negative impact if the assessor
consistently gets the test wrong. A descriptive panel will most likely benefit from feedback to help facilitate learning and skill
improvement. The feedback given during this phase is more specific and relates to the assessor/panel performance results. It helps
to continue the development of the assessors’ skills and performance level.
5.6.4 Panel Monitoring and Maintenance—General feedback on panel performance is good for monitoring assessor performance
over time and maintaining the panel’s proficiency level between projects. Maintenance activities may include conducting
specifically designed tests to evaluate panel/assessor performance; retraining on existing methods and products as deemed
necessary; expanding/enhancing skills required for new methods, attributes, and products; dealing with panel attrition; and so forth.
5.7 Individual Assessor Versus Group Feedback to the Panel as a Whole:
5.7.1 Feedback is important for both individual assessors, who perform the task, and the panel as a whole, who produce the results
used for decision making.
5.7.2 Individual feedback is used as a status check for the individual assessor and provides them with information about their own
performance versus that of the panel. It may also be needed for difficult situations, for example, when the panel leader needs to
draw attention to or correct an assessor’s behavior. It can also be used to motivate an assessor if they have improved their
performance or made a positive contribution to the panel.
5.7.3 Group feedback is given to the panel as a group. It can be used to highlight data performance issues, for example, presenting
plots after a descriptive analysis/profiling project and discussing the attributes that have a lot of disagreement between assessors
and what training or behavior changes will be needed to improve data quality. It can also be a form of motivation, for example,
having a senior manager thank the panel for their contribution to a project.
5.8 Types of Feedback:
5.8.1 There are five main types of feedback that can be used with assessors and panels. Each type of feedback may be used for
different tasks during the different phases of panel development and each has advantages and disadvantages.
5.8.1.1 Correct Response—Informs the assessor of the correct answer.
5.8.1.2 Confirmation Feedback—Informs the assessor whether a response was correct or incorrect. The correct response is not
given to them.
5.8.1.3 Immediate Feedback (within a session)—Information provided after the assessor gives their response; it is most beneficial
when received during training. Immediate feedback can be given directly after the assessor gives a response (refer to Case Study
4, Appendix X4) or it can be given very soon after the assessor’s response is entered during the same panel session. Immediate
feedback can be used during training, but is not recommended during data collection as this can bias the test results.
Findlay, C. J., Castura, J. C., and Lesschaeve, I., “Feedback calibration: A training method for descriptive panels,” Food Quality and Preference, Vol 18, No. 2, 2007,
pp. 321-328, ISSN 0950-3293, https://doi.org/10.1016/j.foodqual.2006.02.007.
E3313 − 24
5.8.1.4 Delayed Feedback—Often, it is not possible to give feedback immediately. The data may require analyzing, plots prepared,
and feedback planned. Be aware that learning may be reduced, especially with longer gaps between the activity and the feedback.
5.8.2 Tables 1-3 summarize which type of performance feedback you can use during each type of panel and phase of panel
development and also provide some examples.
6. Feedback During Descriptive Panel Training
6.1 Descriptive panels profile products by quantifying attribute intensities on rating scales. They are usually trained in specific
profiling methods and familiarized with specific products or product categories. These panels should demonstrate good
repeatability of results. Attrition is an issue for longstanding panels, requiring integration of new members on occasion. To
maintain high-performing panels, training is an ongoing effort as is performance monitoring (with feedback).
6.2 Feedback During Training of New Assessors:
6.2.1 For brand new assessors without previous descriptive analysis experience, it is critical for them to understand that many
of the skills needed to become a trained assessor on a descriptive panel are foreign to normal product usage patterns, but with
training, they will learn to grasp the concepts and apply this new knowledge to the evaluation process. They should be encouraged
to ask questions at each session and understand that they will make mistakes during this phase of development. In general, training
should commence with simple tasks then proceed to more complicated work as the assessors learn. Continuous feedback on each
TABLE 1 Types of Feedback Given during Different Phases of a Descriptive Panel
Phase Types of Appropriate Feedback Examples
Training of new assessors Confirmation response Telling assessors that they had the incorrect
response for a four-sample sweet ranking test.
Correct response Giving assessors the correct order of samples in a
four-sample sweet ranking test.
Immediate feedback (within a session) Telling the assessor who got the sweet ranking test
to go back and taste the samples in the sweet
ranking test and to focus on how they ranked the
less intense samples.
Praising an assessor for accomplishing a new task
(motivation).
Requesting more explanation from an assessor
when they are struggling with a new attribute or
task.
Asking assessors to talk one at a time during panel
discussion (behavior change).
Delayed feedback Discussing any changes in scoring behaviors that
might be needed after reviewing a plot of mean
scores and standard deviation for each assessor
versus the panel.
Giving positive feedback on performance
improvement for a difficult attribute when viewing the
plot.
Project or product category training Immediate feedback (within a session) Confirming that an assessor has scored the intensity
of an attribute close to the desired target or to the
panel average.
Asking assessor to adjust their scores if using the
consensus method.
Requesting an assessor to review and adjust their
attribute scores during a group training session.
Delayed feedback Discussing any changes in scoring behaviors that
might be needed after reviewing a plot of mean
scores and standard deviation for each assessor
versus the panel.
Post-product assessment and data collection Delayed feedback Praising good performance of specific assessors
versus the panel when showing a plot of attribute
mean scores and standard deviations to the panel.
Highlighting attributes with poor performance that
will need more training.
Panel monitoring and maintenance Delayed feedback Informing individual assessors of the attributes they
are not performing well on during the one-year
review of results from a two-year storage trial.
E3313 − 24
TABLE 2 Types of Feedback Given During Different Phases of a
Discrimination Panel
Phase Discrimination Examples
Training of new Correct feedback Assessor is told they
assessors answered a tetrad test
incorrectly and shown
the correct groupings.
Confirmation feedback Assessor is told they
did not get a triangle
test correct and is not
told what the correct
answer was.
Immediate feedback Assessor is told they
(within a session) should reassess the
triangle test samples
and focus on the bitter
taste in the samples.
Project or product Correct feedback Assessor is told they
category training answered a triangle test
incorrectly and shown
the correct groupings.
Immediate feedback Panel leader praises an
assessor for getting
three triangle tests
correct in the training
session.
Post-product Correct feedback Assessor is told they
assessment and data answered the tetrad
collection test correctly.
Panel monitoring and Delayed feedback Assessors are shown a
maintenance count of how many
triangle tests they have
participated in over the
past month and given a
gift voucher if they
attended all ten tests.
new task is important to the learning process. Assessors should be made aware of the “correct” response, where applicable/relevant,
and should re-experience test sample(s) so that they can align their perception to the expected response. The “correct” response
can be determined by past panel work within the company, literature, or the current panel once it is more experienced.
6.2.2 Immediate feedback during initial training can help to ensure poor habits are corrected early. Examples of poor habits include
not following protocol(s); not allowing adequate time between samples; adopting misinterpretation or misuse of scale methods;
habitual, non-discriminating use of scales; and misunderstanding and incorrect rating of specific attributes.
6.3 Feedback During Project Training:
6.3.1 During project training, two-way communication between the panel leader and assessors should be open and frequently
encouraged. The panel may debate and discuss as they try to arrive at common understanding and potentially resolve differences
for new sample sets or when learning new methods or attributes.
6.3.2 During training sessions, assessors commonly assess samples and references during group discussions. Various tools can be
used to determine what feedback to give and supplement feedback. The following formats are commonly used:
6.3.2.1 Attribute discrimination (good versus poor discrimination attribute lists),
6.3.2.2 Graphical illustrations (including boxplots, mean and target ranges, interaction plots, and so forth),
6.3.2.3 Reports or presentations, or both (often providing a historical overview and depiction of performance over time),
6.3.2.4 Targets/references (panel leader provides qualitative or quantitative examples, or both),
6.3.2.5 Correct answer or agreement, and
E3313 − 24
TABLE 3 Types of Feedback Given during Different Phases of a
Quality Panel
Phase Quality Examples
Training of new Correct feedback Assessor is told they
assessors correctly identified the
sample spiked with
butyric acid.
Confirmation feedback Telling assessors that
they had the incorrect
response for a four-
sample ranking test.
Immediate feedback Praising an assessor
for accomplishing a
new task (motivation).
Project or product Correct feedback Assessor is told they
category training answered a triangle test
incorrectly and shown
the correct groupings.
Immediate feedback Panel leader praises an
assessor for correctly
identifying and
recognizing the spiked
samples in a training
session.
Post-product Delayed feedback Project leader shares
assessment and data business outcome from
collection the panel evaluation.
Panel monitoring and Immediate feedback After a “true-to-type”
maintenance monitoring test,
assessors are told the
correct descriptions and
allowed to reassess the
samples.
Delayed feedback Panel is informed of the
attributes they need
further training on after
reviewing results of
daily evaluations (see
case study for more
information).
6.3.2.6 Individual assessor rank compared to panel as a whole.
6.3.3 Feedback about assessor performance can occur after collecting individual raw scores from each assessor, either on paper
or electronically. The panel leader can provide means, ranges, and other statistical analyses, including graphs, for the discussion
with the panel. This process will allow for the assessors to see how they perform individually and, in comparison, with the rest
of the panel.
6.3.4 Based on the mean and the range of scores obtained for each attribute, the panel can discuss why they agree or disagree with
the average rating. The panel should re-experience the product, assessors should determine if they would like to adjust their scores,
and the panel should come to agreement about whether the mean accurately reflects the product attribute under discussion.
Continuous feedback from the panel leader is important to facilitating this cyclic process.
6.3.5 Individual Assessor Feedback—Assessors should understand the importance of the data they give to the project and how
that influences decisions. The panel leader should consider the assessors’ background knowledge when sharing performance tables
or graphs. Make sure they understand the terms (for example, mean and standard deviation) and how to read the graphs used to
represent data. Panel leaders need to explain what the assessor should get from the information and how they can use it to improve
their performance. Examples of good and poor quality performances should be pointed out and discussions with the assessor can
identify and resolve issues in evaluating attributes of concern.
6.3.6 Panel Feedback—Panel performance results can be shared to both inform and motivate panels, though make sure to consider
the background of the panel and project to determine if sharing such results is appropriate. An overview of satisfactory and
unsatisfactory performances should precede sharing of specific results so that members of the panel can focus their attention where
needed. Also, the level of expectations for performance should be described. Of particular interest to panel members are the
E3313 − 24
following questions: (1) how did the panel do overall, both positive and constructive examples (2) how can the panel improve, (3)
how did the individual assessor do overall, both positive and constructive examples, and (4) where can individual assessor
improvements be made.
6.4 Feedback After Data Collection—The same tools and type of feedback given to assessors and the panel during training (6.3)
can also be used after data collection is complete. Giving this type of feedback is dependent on any effect that it may have on the
project objectives and final results. For example, information shared after a one-off project may be different from that giving during
an ongoing study such as a shelf-life study.
6.5 Feedback During Panel Monitoring—Panel maintenance involves preserving the performance level of the panel for continued
use. Training exercises are designed to ensure the panel does not drift and that they maintain their ability to discriminate at the
desired level of sensitivity needed to make business decisions. Activities may include conducting specifically designed tests to
evaluate panel/assessor performance; retraining on existing methods and products; and expanding/enhancing skills required for
new methods, attributes, and products.
6.5.1 Individual Assessor Feedback:
6.5.1.1 Similar feedback can be given as described in 6.3.5. If an assessor’s performance deteriorates over time, it will be
necessary to give them repeated feedback on the areas in which they need to improve and how to improve. Make sure to consider
the information in 5.7 when having this type of discussion.
6.5.1.2 If performance does not improve, it may eventually become necessary to remove the assessor from the panel. If applicable,
consult and involve Human Resources when planning to have this type of discussion with an assessor.
6.5.2 Panel Feedback—Similar feedback can be given to the panel as described in 6.3.6.
7. Feedback During Discrimination Testing
7.1 In a discrimination test, the assessor’s task is to determine if the products that they are presented with are different from each
other. There are many different discrimination tests available depending on the test objectives, product types, and resources
available, but the feedback approach for screened and trained assessors is quite similar for all methods.
7.2 Discrimination panels are typically company employees or trained panels; these are assessors who continuously participate on
this type of panel. Feedback plays an important role in motivating them to return for subsequent tests. The feedback for overall
or unspecified discrimination tests (for example, same-different or triangle) differs slightly from that for attribute-specific
discrimination tests (for example, directional paired comparison or two alternative forced choice) as described in 7.3.
7.3 Feedback During Training of New Assessors:
7.3.1 When training new assessors in any discrimination task, the main feedback objectives are to ensure that the assessor knows
how to take part in the test and instill confidence in the assessors for subsequent real test situations. The products used for training
are generally known to be different and the objective is to qualify the assessor for future testing situations for that discrimination
method, for example, duo-trio or tetrad.
7.3.2 In attribute-specific discrimination tests, a further objective is added to the training: ensuring that the assessor understands
and recognizes the attribute of interest. For example, if the two-alternative forced choice training test is to determine which of two
beers is more bitter, the assessor shall be able to understand that they are looking for a difference in bitterness and not, say,
carbonation or sourness, but also that they can detect and quantify the levels of bitterness in the beers to be able to recognize the
difference in the samples presented. Training and feedback for attribute-specific discrimination tests, therefore, also include tests
to help the assessor recognize and quantify the attribute of interest by gradually decreasing the level of difference in the attribute.
7.3.3 When training new assessors in discrimination tests, it can be very helpful to give immediate feedback to the assessor on
their performance so that they may reassess the products, if possible, to confirm or correct their result. Replication of the training
test(s) on different occasions, again with immediate feedback, can improve the assessor’s ability in that discrimination method. For
example, they may learn that they work better in a particular discrimination test if they make their decision based on their initial
judgement or, alternatively, that they need to assess the samples a couple of times to confirm their answer.
E3313 − 24
7.4 Feedback During Project Training:
7.4.1 Immediate feedback to the assessor regarding their “correct response” is only recommended for training tests and not real
test situations. There are several reasons for this. If the sensory facility is mainly conducting similarity tests and an assessor is only
able to identify the odd sample correctly in a series of triangle tests around 33 % of the time, being given constant feedback that
they were “incorrect” may lower motivation to be on the panel. If a sensory facility replicates all discrimination tests, immediate
feedback after each replication can influence the results of the next replicate. Also, when working on a large project such as salt
or sugar reduction, giving assessors immediate feedback about whether they were “correct” or not may well impact their
performance in the next related tests by giving them clues as to the objective of the tests.
7.4.2 However, delayed feedback relating to project progress (without the specific detail about the project) can be very motivating
to the assessors, as well as a simple “thank you for good attendance.”
7.5 Feedback During Panel Monitoring—Discrimination assessors who test on a regular basis should receive feedback on their
performance both as individuals and a group.
7.5.1 Individual Assessor Feedback:
7.5.1.1 Monitoring can be simply performed by recording each assessor’s results over time and comparing to the panel’s result:
in fact, some software systems will offer this as standard. If the panel found a statistically significant difference between the
products, each individual assessor’s result is compared to this. The sensory science function can then set their own action standard
for the performance monitoring such as: “if an assessor does not identify a difference in more than 40 % of tests when the panel
as a whole has found a difference, validation tests (see Note 1) will be performed to check that the assessor still meets the
requirements for the discrimination testing panel.”
NOTE 1—A validation test is a test in which the result is already known and an assessor’s performance can be compared directly to this.
7.5.1.2 Recording the results of all tests will also help identify those assessors who are able to detect consistently small differences
or are particularly adept at taint detection, for example. However, this information is rarely used for feedback except to report and
recognize attendance.
7.5.1.3 When validation tests are performed regularly, informing the assessors if they were “correct” in their sample choice can
be very motivating for the assessors; however, be careful that the differences between the products do not become common
knowledge or further validation study products will need to be sourced.
7.5.2 Panel Feedback—Group feedback for a discrimination panel usually consists of a summary of the number of tests conducted
and a thank you from the panel leader or project leader for the assessors’ contributions. If the panel takes part in any comparative
testing with panels in other locations, sharing the overall outcome of these tests, for example, that the panel managed to identify
the difference as a team, can also be interesting and aspirational for the assessors. After testing is complete, assessors may be
motivated by learning the objective of the test and how their work saved the company money, accomplished a specific objective,
helped to launch a new product, and so forth. However, be careful that the information shared will not bias the assessors for future
projects.
8. Sensory Quality Testing
8.1 Different types of sensory quality panels are used in industry including: quality assurance panels that support assessment of
incoming ingredients and outgoing plant production; shelf-life panels to monitor quality of the product over time and under
different storage conditions; panels dedicated to new product development initiatives, sourcing, and procurement; and other
applications in which product quality is a concern.
8.2 Sensory quality panels are typically comprised of dedicated assessors very familiar with the products being tested. The
assessors can be internal employees of the company or external assessors. Usual/routine evaluations often involve comparison to
controls or retains. As opposed to descriptive panels that involve detailed profiling, the quality assurance (QA) sensory quality
panels generally are tasked to rate overall match to a target or control in a very efficient, rapid manner to keep pace with production
and ensure product sensory specifications are satisfied. As a result, quality panels often test a high volume of products on a daily
E3313 − 24
basis and generally work more hours in a day assessing the sensory aspects of products. Performance expectations should be clearly
communicated to the assessors when they start in their role as this makes marginal or poor performance more easily addressed.
8.3 Typical quality methods may be as simple as discrimination tests (for example, duo-trio or tetrad) and designating a sample
as in or out of specification (“In/Out” or A-not-A) or more diagnostic such as the relative-to-reference (directional-degree-of-
difference) scale, simple descriptive analysis, or more complex shelf-life determination or confirmation. Training and monitoring
performance with feedback involves ensuring assessors are familiar with the control product(s) and the acceptable range of
deviation, although some quality methods do not require the assessment of controls within the test itself or do not have the ability
to keep controls stable. In these instances, assessors refer to a written sensory specification.
8.4 Training and monitoring should also ensure that assessors are capable of determining that the intended attributes or “on notes”
are within the target intensity range and they are also capable of identifying any potential defects of “off notes” of each product.
Therefore, where possible, it is good practice to present products containing both “on” and “off” notes to the quality assessors
during training so that they can experience the full range of characteristics.
8.5 To help assessors understand desirable product attributes, product identification training and feedback can be given. In
training, assessors are presented with a sample whose brand is not given and they shall correctly identify the brand. Assessor
performance on brand identification should be tracked and compared against preset validation parameters (for example, 75 %
identification of each brand).
8.6 Assessors should also be trained and provided feedback on specific off attributes that can occur in products they evaluate.
Typically, this is done by adding a specific off attribute to the product and presenting it to the assessor either blind during normal
assessment or in a training setting in which they are aware doctored samples are present. Both detection, which is the ability of
the assessor to perceive an atypical aspect, and recognition, which is the ability of the assessor to identify correctly the off note,
should be tracked and compared against preset validation parameters as with brand identification.
8.7 Feedback on assessor repeatability is also important. Some methods may already incorporate this type of data analysis, but
others such as in/out should have repeated samples included and are analyzed with a qualitative approach. If an assessor is
consistent in saying a sample is in or out of specification and their comments are the same between the two samples, they would
be considered repeatable.
8.8 Individual assessor responses should also be examined for agreement with the broader panel response. Though certain
methods such as difference from control and descriptive analysis easily allow for such analyses, qualitative methods may be used
for agreement for methods such as in/out. For example, it is straightforward to examine if an assessor indicates samples are out
of specification more often than the panel does on average. If assessors make comments on samples, these too can be compared
qualitatively between a single assessor and the panel, and discrepancies can be given as feedback to individual assessors.
8.9 For all types of quality performance, it may be useful to compare assessor and panel performance in brand identification,
attribute recognition, repeatability, and agreement across multiple production facilities to ensure the same assessor performance,
product(s) characteristics, and quality levels are maintained at all locations.
8.10 Feedback During New Assessor Training—Communication regarding the method, product characteristics, ranges, and so
forth during the training of a sensory quality panel is critical regardless of the type of panel (plant, corporate, or external). Assessors
will require clear instruction on evaluation methodology and familiarization with the ingredients and finished products that are
being evaluated, as well as the “on” and “off” notes. During the training, the assessors should also develop an understanding of
their own abilities, for example, when they are anosmic to certain ingredient or product characteristics or they are especially
sensitive to other aspects such as taints or “off-notes.” The training should allow assessors to understand that everyone has different
sensitivities and the role is as a group, all working together. This helps boost confidence and morale of the whole quality panel.
As well as these physiological limitations, training should also include information about psychological errors; biases such as
expectation, logical, and habitual errors are especially important for all quality assessors to understand so that they can help prevent
issues.
8.11 Feedback During an Ongoing Operation:
8.11.1 Feedback is essential for sensory quality panels, but because of innate differences between a plant, corporate, or external
E3313 − 24
quality panel, the type and delivery of feedback will vary. If working on a project basis (for example, shelf life), feedback on
performance can be given during training on the samples or category. If the panel is working in a continuous production
environment, feedback might be best given immediately to individual assessors, or results may be collected and given as a
summary report at a regular time point (for example, daily or weekly).
8.11.2 The sensory scientist should ensure that feedback does not have a detrimental impact on future testing programs. For
example, if the method of choice is the “in-out” method, the assessors are in control of the decision-making process: if the success
of their results are linked to an annual bonus, this may lead to approval of all products. Compare this to the potentially less biasing
descriptive specification method in which the sensory scientist makes the decision whether a product is “in” or “out” of
specification based on the assessors’ results. For discrimination testing, often the “right” result is not being able to identify correctly
the difference, especially for minor ingredient or production changes, and assessors should not be penalized for their answers.
Assessors should be rewarded for taking part, not just their results.
8.11.3 Individual Assessor Feedback:
8.11.3.1 Assessors should be given feedback on their attendance, success, and motivation to take part in the routine assessments.
A simple “thank you” for attendance can be highly motivating and motivated assessors generally produce better data.
8.11.3.2 Immediate feedback about an assessor’s performance can be given after a test has been completed or if data are
continuously being collected. Feedback can also be given at a regular time point (for example, daily, weekly, or monthly) via
graphs or reports. Feedback on how an assessor is performing over production time and batches is beneficial to ensure consistent
results, although quality panel feedback may not be as regular as descriptive analysis or discrimination panels. Feedback is also
useful as part of the regular retraining and assessment of quality panels, usually conducted several times a year. Poor assessor
performance may result in variable assessments (different scores given for the same repeated product in a series), wrongly
identifying the attribute (confusing typical “on” or “off” notes) or not discriminating between samples the rest of the assessors
discriminated (maybe due to assessor habituation error).
8.11.3.3 To monitor and help prevent these issues, regular “out” samples or test replicates should be used to validate assessor
performance on a daily basis (see 8.12).
8.11.3.4 If the quality project involves descriptive analysis or descriptive analysis-type scaling approaches, please see Section 6
for more information about feedback for these methods. For discrimination tests (for example, difference from control and ranking)
see Section 7. For specific quality methods such as “in-out of specification,” there are two main types of feedback: the results of
an individual assessor can be compared to the rest of the panel for a specific test or an individual’s results can be monitored over
time. For the latter, please see 8.12.
8.11.4 Panel Feedback—The same approach mentioned in 8.11.3 can be used for panel feedback. Feedback can be given to a panel
at the end of a discrete project or in a production environment regular feedback on performance over time, and product batches
should be scheduled in as part of morning panels or weekly/monthly trainings.
8.12 Feedback During Panel Monitoring and Maintenance:
8.12.1 Quality panels typically become very familiar with the incoming ingredients and outgoing finished goods from production
and generate substantial amounts of data that can be tracked over time, for example
...