SIST EN ISO 11132:2017
(Main)Sensory analysis - Methodology - Guidelines for monitoring the performance of a quantitative sensory panel (ISO 11132:2012)
Sensory analysis - Methodology - Guidelines for monitoring the performance of a quantitative sensory panel (ISO 11132:2012)
ISO 11132:2012 gives guidelines for monitoring and assessing the overall performance of a quantitative descriptive panel and the performance of each member.
A panel of assessors can be used as an instrument to assess the magnitude of sensory attributes.
Performance is the measure of the ability of a panel or an assessor to make valid attribute assessments across the products being evaluated. It can be monitored at a given time point or tracked over time. Performance comprises the ability of a panel to detect, identify, and measure an attribute, use attributes in a similar way to other panels or assessors, discriminate between stimuli, use a scale properly, repeat their own results, and reproduce results from other panels or assessors.
The methods specified allow the consistency, repeatability, freedom from bias and ability to discriminate of panels and assessors to be monitored and assessed. Monitoring and assessment of agreement between panel members is also covered. Monitoring and assessment can be carried out in one session or over time.
Monitoring performance data enables the panel leader to improve panel and assessor performance, to identify issues and retraining needs or to identify assessors who are not performing well enough to continue participating.
The methods specified in ISO 11132:2012 can be used by the panel leader to appraise continuously the performance of panels or individual assessors.
ISO 11132:2012 applies to individuals or panels in training as well as for established panels.
Sensorische Analyse - Methodologie - Leitlinien zur Überprüfung der Leistungsfähigkeit eines quantitativen sensorischen Panels (ISO 11132:2012)
Analyse sensorielle - Méthodologie - Lignes directrices pour le contrôle de la performance d'un jury sensoriel quantitatif (ISO 11132:2012)
L'ISO 11132:2012 donne des lignes directrices pour contrôler et évaluer la performance globale d'un jury descriptif quantitatif et la performance de chaque membre.
Un jury de sujets peut être utilisé comme un instrument pour évaluer l'intensité d'attributs sensoriels.
La performance est la mesure de la capacité d'un jury ou d'un sujet à effectuer des jugements valides d'attributs concernant les produits à évaluer. Elle peut être contrôlée à un moment donné ou suivie au fil du temps. La performance est la capacité d'un jury à détecter, identifier et mesurer un attribut, à utiliser les attributs de la même manière que d'autres jurys ou sujets, à différencier les stimulus, à utiliser correctement une échelle, à répéter ses propres résultats et à reproduire les résultats d'autres jurys ou sujets.
Les méthodes spécifiées permettent de contrôler et d'évaluer la cohérence, la répétabilité, l'absence de biais et la capacité de discrimination de jurys et de sujets. Le contrôle et l'évaluation de la concordance entre les membres du jury sont également abordés. Le contrôle et l'évaluation peuvent être réalisés au cours d'une session ou au fil du temps.
Le contrôle des données de performance permet au responsable du jury d'améliorer la performance du jury et des sujets, d'identifier les problèmes et les besoins de remise à niveau ou d'identifier les sujets qui ne sont pas suffisamment performants pour continuer à participer.
Les méthodes spécifiées dans l'ISO 11132:2012 peuvent être utilisées par le responsable du jury pour évaluer de façon continue la performance du jury ou de chaque sujet.
L'ISO 11132:2012 s'applique à des sujets ou à des jurys en formation aussi bien qu'à des jurys établis.
Senzorična analiza - Metodologija - Smernice za nadzorovanje izvajanja kvantitativnega senzoričnega panelnega testa (ISO 11132:2012)
Standard ISO 11132:2012 določa smernice za nadzorovanje in ocenjevanje splošne učinkovitosti izvajanja kvantitativnega opisnega panelnega preskusa in izvajanje vsakega člana.
Panel ocenjevalcev je mogoče uporabiti kot instrument za ocenjevanje magnitude senzoričnih atributov.
Izvajanje je merilo za zmožnost panela ali ocenjevalca, da veljavno oceni atribute vrednotenih izdelkov. Spremljati ga je mogoče v dani časovni točki ali v določenem časovnem obdobju. Izvajanje zajema sposobnost panela, da zazna, prepozna in izmeri atribut, uporablja atribute na podoben način kot drugi paneli ali ocenjevalci, razlikuje dražljaje, pravilno uporablja lestvico, ponovi svoje rezultate in poustvari rezultate drugih panelov ali ocenjevalcev.
Navedene metode omogočajo doslednost, ponovljivost, nepristranskost in sposobnost razlikovanja panelov in ocenjevalcev, ki jih je treba nadzorovati in oceniti. Zajeto je tudi nadzorovanje in ocenjevanje dogovora med člani panela. Nadzorovanje in ocenjevanje je mogoče opraviti v eni seji ali določenem časovnem obdobju.
Nadzorovanje podatkov o izvajanju omogoča vodji panela, da izboljša panel in izvajanje ocenjevalca, da prepozna težave in potrebe po ponovnem usposabljanju ter da prepozna ocenjevalce, katerih izvajanje ni zadostno za nadaljnje sodelovanje.
Z metodami, določenimi v standardu ISO 11132:2012, lahko vodja panela nenehno ocenjuje izvajanje panelov ali posameznih ocenjevalcev.
ISO 11132:2012 se uporablja za posameznike ali panele med usposabljanjem ter za uveljavljene panele.
General Information
Relations
Standards Content (Sample)
SLOVENSKI STANDARD
SIST EN ISO 11132:2017
01-september-2017
6HQ]RULþQDDQDOL]D0HWRGRORJLMD6PHUQLFH]DQDG]RURYDQMHL]YDMDQMD
NYDQWLWDWLYQHJDVHQ]RULþQHJDSDQHOQHJDWHVWD ,62
Sensory analysis - Methodology - Guidelines for monitoring the performance of a
quantitative sensory panel (ISO 11132:2012)
Sensorische Analyse - Methodologie - Leitlinien zur Überprüfung der Leistungsfähigkeit
eines quantitativen sensorischen Panels (ISO 11132:2012)
Analyse sensorielle - Méthodologie - Lignes directrices pour le contrôle de la
performance d'un jury sensoriel quantitatif (ISO 11132:2012)
Ta slovenski standard je istoveten z: EN ISO 11132:2017
ICS:
67.240 6HQ]RULþQDDQDOL]D Sensory analysis
SIST EN ISO 11132:2017 en
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
---------------------- Page: 1 ----------------------
SIST EN ISO 11132:2017
---------------------- Page: 2 ----------------------
SIST EN ISO 11132:2017
EN ISO 11132
EUROPEAN STANDARD
NORME EUROPÉENNE
May 2017
EUROPÄISCHE NORM
ICS 67.240
English Version
Sensory analysis - Methodology - Guidelines for
monitoring the performance of a quantitative sensory
panel (ISO 11132:2012)
Analyse sensorielle - Méthodologie - Lignes directrices Sensorische Analyse - Methodologie - Leitlinien zur
pour le contrôle de la performance d'un jury sensoriel Überprüfung der Leistungsfähigkeit eines
quantitatif (ISO 11132:2012) quantitativen sensorischen Panels (ISO 11132:2012)
This European Standard was approved by CEN on 31 January 2017.
CEN members are bound to comply with the CEN/CENELEC Internal Regulations which stipulate the conditions for giving this
European Standard the status of a national standard without any alteration. Up-to-date lists and bibliographical references
concerning such national standards may be obtained on application to the CEN-CENELEC Management Centre or to any CEN
member.
This European Standard exists in three official versions (English, French, German). A version in any other language made by
translation under the responsibility of a CEN member into its own language and notified to the CEN-CENELEC Management
Centre has the same status as the official versions.
CEN members are the national standards bodies of Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia,
Finland, Former Yugoslav Republic of Macedonia, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania,
Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland,
Turkey and United Kingdom.
EUROPEAN COMMITTEE FOR STANDARDIZATION
COMITÉ EUROPÉEN DE NORMALISATION
EUROPÄISCHES KOMITEE FÜR NORMUNG
CEN-CENELEC Management Centre: Avenue Marnix 17, B-1000 Brussels
© 2017 CEN All rights of exploitation in any form and by any means reserved Ref. No. EN ISO 11132:2017 E
worldwide for CEN national Members.
---------------------- Page: 3 ----------------------
SIST EN ISO 11132:2017
EN ISO 11132:2017 (E)
Contents Page
European foreword . 3
2
---------------------- Page: 4 ----------------------
SIST EN ISO 11132:2017
EN ISO 11132:2017 (E)
European foreword
The text of ISO 11132:2012 has been prepared by Technical Committee ISO/TC 34 “Food products” of
the International Organization for Standardization (ISO) and has been taken over as
EN ISO 11132:2017.
This European Standard shall be given the status of a national standard, either by publication of an
identical text or by endorsement, at the latest by November 2017, and conflicting national standards
shall be withdrawn at the latest by November 2017.
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. CEN [and/or CENELEC] shall not be held responsible for identifying any or all such patent
rights.
According to the CEN-CENELEC Internal Regulations, the national standards organizations of the
following countries are bound to implement this European Standard: Austria, Belgium, Bulgaria,
Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, Former Yugoslav Republic of Macedonia,
France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta,
Netherlands, Norway, Poland, Portugal, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland,
Turkey and the United Kingdom.
Endorsement notice
The text of ISO 11132:2012 has been approved by CEN as EN ISO 11132:2017 without any modification.
3
---------------------- Page: 5 ----------------------
SIST EN ISO 11132:2017
---------------------- Page: 6 ----------------------
SIST EN ISO 11132:2017
INTERNATIONAL ISO
STANDARD 11132
First edition
2012-11-01
Sensory analysis — Methodology —
Guidelines for monitoring the performance
of a quantitative sensory panel
Analyse sensorielle — Méthodologie — Lignes directrices pour le
contrôle de la performance d’un jury sensoriel quantitatif
Reference number
ISO 11132:2012(E)
©
ISO 2012
---------------------- Page: 7 ----------------------
SIST EN ISO 11132:2017
ISO 11132:2012(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO 2012
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO’s
member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO 2012 – All rights reserved
---------------------- Page: 8 ----------------------
SIST EN ISO 11132:2017
ISO 11132:2012(E)
Contents Page
Foreword .iv
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Principle . 2
5 Experimental conditions . 4
6 Qualification of assessors . 4
7 Procedure . 4
7.1 Monitoring via formal performance validation . 4
7.2 Statistical analysis of data from formal performance validation (a single session) . 5
7.3 Overall panel performance from formal performance validation . 6
7.4 Individual assessor performance from formal performance validation . 7
7.5 Performance issues . 9
7.6 Monitoring via routine product profiling . 9
7.7 Experimental design for study of performance over time . 9
7.8 Statistical analysis of data over time . 9
7.9 Reproducibility between panels .10
7.10 Statistical analysis of complete profiles .10
Annex A (informative) Example of practical application . 11
Annex B (informative) Example of use of cusum analysis .18
Annex C (informative) Example of use of Shewhart chart .21
Bibliography .23
© ISO 2012 – All rights reserved iii
---------------------- Page: 9 ----------------------
SIST EN ISO 11132:2017
ISO 11132:2012(E)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies
(ISO member bodies). The work of preparing International Standards is normally carried out through ISO
technical committees. Each member body interested in a subject for which a technical committee has been
established has the right to be represented on that committee. International organizations, governmental and
non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the International
Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of technical committees is to prepare International Standards. Draft International Standards
adopted by the technical committees are circulated to the member bodies for voting. Publication as an
International Standard requires approval by at least 75 % of the member bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO shall not be held responsible for identifying any or all such patent rights.
ISO 11132 was prepared by Technical Committee ISO/TC 34, Food products, Subcommittee SC 12,
Sensory analysis.
iv © ISO 2012 – All rights reserved
---------------------- Page: 10 ----------------------
SIST EN ISO 11132:2017
INTERNATIONAL STANDARD ISO 11132:2012(E)
Sensory analysis — Methodology — Guidelines for monitoring
the performance of a quantitative sensory panel
1 Scope
This International Standard gives guidelines for monitoring and assessing the overall performance of a
quantitative descriptive panel and the performance of each member.
A panel of assessors can be used as an instrument to assess the magnitude of sensory attributes.
Performance is the measure of the ability of a panel or an assessor to make valid attribute assessments across
the products being evaluated. It can be monitored at a given time point or tracked over time. Performance
comprises the ability of a panel to detect, identify, and measure an attribute, use attributes in a similar way to
other panels or assessors, discriminate between stimuli, use a scale properly, repeat their own results, and
reproduce results from other panels or assessors.
The methods specified allow the consistency, repeatability, freedom from bias and ability to discriminate of
panels and assessors to be monitored and assessed. Monitoring and assessment of agreement between panel
members is also covered. Monitoring and assessment can be carried out in one session or over time.
Monitoring performance data enables the panel leader to improve panel and assessor performance, to
identify issues and retraining needs or to identify assessors who are not performing well enough to continue
participating.
The methods specified in this International Standard can be used by the panel leader to appraise continuously
the performance of panels or individual assessors.
This International Standard applies to individuals or panels in training as well as for established panels.
2 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced document
(including any amendments) applies.
ISO 5492, Sensory analysis — Vocabulary
ISO 8586, Sensory analysis — General guidelines for the selection, training and monitoring of selected and
expert assessors
ISO 8589, Sensory analysis — General guidance for the design of test rooms
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO 5492 and the following apply.
3.1
agreement
ability of different panels or assessors to assign similar scores on a given attribute to samples of the same product
3.2
homogeneity
measure of the agreement of responses among individual assessors within a test session, as a panel of
assessors in replicate sessions, or for an individual assessor in replicate sessions
© ISO 2012 – All rights reserved 1
---------------------- Page: 11 ----------------------
SIST EN ISO 11132:2017
ISO 11132:2012(E)
3.3
assessor bias
tendency of an assessor to give scores which are consistently above or below the true score when that is
known or the panel mean when it is not
3.4
outlier
an assessment that does not conform to the overall pattern of the data or is extremely different from other
assessments of the same or similar products
3.5
panel drift
phenomenon where a panel, over time, changes in sensitivity or becomes susceptible to biases and as a
consequence changes the location on the scale where an attribute is rated for a constant, reference product
3.6
performance
ability of a panel or an assessor to make valid and reliable assessments of stimuli and stimulus attributes
3.7
repeatability
agreement in assessments of equivalent product samples under the same test conditions by the same
assessor or panel
3.8
reproducibility
agreement in assessments of equivalent product samples under different test conditions, with different tasks
or by a different assessor or panel
NOTE Reproducibility may be measured as any of the following:
— the reproducibility of a panel in the short term, measured between two or more sessions separated by several days;
— the reproducibility of a panel in the medium or long term, measured among sessions separated by several months;
— the reproducibility between different panels, in the same laboratory or in different laboratories;
— the reproducibility of assessments by a single assessor of different attributes of a product.
3.9
validation
process of establishing that sensory data correlate with other data on samples of the same product (e.g.
laboratory measurements, consumer perception, results from other panels, consumer complaints) or that a
panel or assessor is able to meet specified performance criteria
3.10
session
occasion on which products are assessed
NOTE In a single session either one or several products may be assessed by one or several assessors. For an
assessor, whether alone or as part of a panel, sessions are separated in time.
3.11
replicate sessions
sessions in which the assessors, the products, the test conditions, and the task are the same
4 Principle
This International Standard is concerned with sensory panels used to assess the magnitude of one or more
sensory attributes in order to make quantitative descriptions or profiles of products. Different methods are
appropriate to the assessment and monitoring of the performance of panels used for difference testing.
2 © ISO 2012 – All rights reserved
---------------------- Page: 12 ----------------------
SIST EN ISO 11132:2017
ISO 11132:2012(E)
The performance of a quantitative sensory panel may be evaluated by using assessments already available or
from panel sessions conducted specifically for the purpose of obtaining performance data.
This International Standard may be used either for periodic monitoring or for reviewing ongoing profile data.
A dedicated monitoring procedure at periodic intervals is appropriate for accreditation and other purposes.
Figure 1 is a flow chart for this procedure.
To review ongoing profile data generated by a panel, it can be appropriate to use data that originated from quite
different profiling experiments using different product types, product numbers, etc. The procedure is the same
as that shown in Figure 1. However, as there are no predefined differences, it is recommended that attributes
that are significantly discriminated by the panel as a whole for a given profile be used as the key measures
to check the performance of individual panelists. Attributes that result in no significant difference cannot be
reliably used to check consistency since the lack of agreement within and between panelists probably means
that the products are very similar for those characteristics.
a) Monitoring by means of performance validation.
Use a small set of samples (perhaps three or four) for which some attributes are known to be different.
These attributes are then used as the key measures on which to measure performance.
↓
b) Overall panel performance
1) How many of the expected key attributes have been significantly discriminated?
2) How many of the key attributes show an interaction of sample and assessor? This gives an initial
indication of where there is least consistency across the panel (7.3.2)
3) Repeatability of the panel for the key attributes in replicate sessions (7.3.3)?
↓
c) Individual assessor performance
1) Discrimination ability: how many of the expected key attributes have been significantly discriminated?
2) Repeatability: consistency of discrimination for a given attribute and product (7.4.2)?
3) Contributions to interaction: for which attributes do interactions occur?
i) Interaction due to cross-over effects (7.4.4)
ii) Interaction due to different use of the scale (7.4.5)
↓
d) Where performance issues have been identified, either for the panel or for individual assessors,
appropriate training sessions should be planned.
Figure 1 — Flow chart for performance monitoring
In a single session, the following indicators can be determined.
— Bias of an assessor, measured as the difference between the assessor’s mean and a known, ‘true’ value,
or the mean of the panel as an estimate of the ‘true’ value.
— Repeatability of an assessor, inversely related to the standard deviation (SD) of repeat assessments by the
assessor of the same sample, or between replicates of the same product.
— Reproducibility of an assessor, inversely related to the SD of the assessor’s biases across individual products.
© ISO 2012 – All rights reserved 3
---------------------- Page: 13 ----------------------
SIST EN ISO 11132:2017
ISO 11132:2012(E)
— Discrimination of an assessor, measured as the ability to assign consistently different scores to
different products.
Bias in an assessor may indicate sensory sensitivity that is different from other assessors and/or use of the
response scale in a way that differs from other assessors.
If an assessor appears to give assessments that differ from those of other assessors, review all the results with
a view to determining whether:
a) the assessments are consistent or variable for repeated samples of the same product;
b) the assessments are similar or different for samples of different products;
c) bias occurs with all, or only some, assessment scales.
Analysis of variance (ANOVA) can be used to investigate these questions.
In some cases, bias may indicate an assessor of superior ability whose results are particularly useful. In other
cases, an assessor showing bias may require retraining or removal from the panel.
A single, consistent approach to statistical analysis of the results is described here. However, some attributes
of panel performance can be assessed by more than one descriptive measure. For instance, error mean
square and error SD (its square root) both express variability in the evaluation of a product. The measures used
should be those that are usual in the field of application.
Other relevant measures of agreement between assessors in the use of the scale for an attribute are the
interaction of assessor and product and the coefficient of correlation between an assessor’s scores and the
panel means. An assessor may have no bias, but may be using the scale in a different way. A correlation close
to 1, a regression slope close to 1, and a regression intercept close to 0 indicate good agreement between an
assessor and the rest of the panel.
With a small number of assessments (fewer than six) the correlation coefficient should be interpreted with
caution, as it can be high (up to 0,7), by chance alone.
5 Experimental conditions
The test facilities shall be in accordance with ISO 8589.
6 Qualification of assessors
The panel shall have the level of qualification and experience of selected assessors (ISO 8586) or better.
7 Procedure
7.1 Monitoring via formal performance validation
At each session, the panel of assessors should be presented with a set of samples similar to those the panel
are to assess when evaluating products and for which statistically significant differences between at least one
pair of the samples can be guaranteed for at least eight attributes.
This number is recommended to encourage panel leaders or sensory managers to identify and select validation
samples that show a realistic as well as a statistical measure of a panel’s performance.
These key attributes are used as key measures against which to assess panel performance. The sample
set should include replicates. There shall be the same number of replicates of each sample. The numbers of
assessors, samples, and replicates depends on the products, the sensory attributes assessed and the purpose
of the procedure. For example 2 or 3, replicates of three or four samples might be used. Care should be taken
to limit the number of assessments required so as to avoid sensory fatigue. The attributes of the samples
should be similar to the range of values that the panel assesses when evaluating products.
4 © ISO 2012 – All rights reserved
---------------------- Page: 14 ----------------------
SIST EN ISO 11132:2017
ISO 11132:2012(E)
A randomized block experimental design has been adopted, in which the assessors are the “blocks”.
If there is expected to be a carry-over effect from one sample to the next, a suitable experimental design is the
Williams Latin square. The basic design uses four assessors and four samples.
Table 1 — Williams Latin square
Order
Assessor
1 2 3 4
1 A B C D
2 B D A C
3 C A D B
4 D C B A
In this design, each assessor samples the four products in a different order and any particular product is
followed by a different one for each assessor, for example A is followed by B for assessor 1, C for assessor 2,
D for assessor 3 and none for assessor 4.
If multiples of four assessors are available, the same design can be repeated for each set of four.
7.2 Statistical analysis of data from formal performance validation (a single session)
Table 2 illustrates one way to tabulate and summarize the results. Some computer software may require a
different organization of the data, for instance with the samples in columns and the assessors in rows.
Table 2 — Results of the assessors
Assessor
Sample 1 2 j n Mean
q
Scores Mean Scores Mean Scores Mean Scores Mean
Y Y
111 1j1
Y Y
112 1j2
Y
Y Y
1
1.j
11. 1.
Y Y
11n 1jn
r r
2
Y Y
i11
ij1
Y
Y
i12 ij2
Y
Y Y
i
ij.
i1. i.
Y Y
in1 ijn
r r
n
p
Y
Y
Mean
..j
...
In this table it is assumed that there are:
n ≡ number of samples (i = 1,2 … n );
p p
n ≡ number of assessors (j = 1,2 … n );
q q
n ≡ number of replicates per sample (k = 1,2 … n ).
r r
Measures of the performance of the panel as a whole and individual assessors, other than bias, require the
data to be analysed by ANOVA.
The details of the basic calculations are not shown in this International Standard, since the analyses are
normally carried out by a computer package.
© ISO 2012 – All rights reserved 5
---------------------- Page: 15 ----------------------
SIST EN ISO 11132:2017
ISO 11132:2012(E)
Each assessor’s data are analysed by one-way ANOVA (Table 3).
Table 3 — ANOVA for an individual assessor for one attribute
Source of variation Degrees of freedom Sum of squares Mean square F-ratio
Between samples ν = n - 1 S MS = s /ν
1 p 1 1 1 1
F = MS /MS
1 2
Error ν = n (n - 1) S MS = s /ν
2 p r 2 2 2 2
Total ν = n n - 1 S
3 p r 3
n ≡ number of samples
p
n ≡ number of replicates per sample
r
The data for the complete session are analysed by randomized block ANOVA (Table 4).
Table 4 — ANOVA for a complete session for one attribute
Source of variation Degrees of freedom Sum of squares Mean square F-ratio
Between samples ν = n - 1 S MS = s /ν
4 p 4 4 4 4
a
Between assessors ν = n - 1 S MS = s /ν F = MS /MS
5 q 5 5 5 5 5 7
Interaction ν = (n - 1)(n - 1) S MS = s /ν F = MS /MS
6 p q 6 6 6 6 6 7
Error ν = n n (n - 1) S MS = s /ν
7 p q r 7 7 7 7
Total ν = n n n - 1 S
8 p q r 8
n ≡ number of samples
p
n ≡ number of assessors
q
n ≡ number of replicates per sample
r
a
If the interaction is significant, the F-ratio for between assessors is calculated by F = MS /MS with the interaction mean square in
5 6
the denominator.
7.3 Overall panel performance from formal performance validation
7.3.1 Key attribute discrimination
The proportion of key attributes that have been significantly discriminated as expected should be determined.
For each attribute, this is indicated by significant variation between samples at a level of 0,05 in the ANOVA
table for a session (Table 4). The higher the proportion of key attributes significantly discriminated, the better
the panel is performing. The panel should receive further training on key attributes that are not significantly
discriminated as expected.
7.3.2 Homogeneity of the panel
A panel is not homogeneous when any assessors are in disagreement with the rest of the panel.
A panel is not homogeneous if the interaction of sample and assessor in the ANOVA is significant at a level of 0,05.
The degree of homogeneity of the panel is inversely related to the interaction SD, s .
i
MS −MS
67
s =
i
n
r
See Table 4.
The number of key attributes giving significant interaction of sample and assessor should be determined. Refer
to the ANOVA table for each attribute and note those showing interaction at a level of 0,05. The higher the
6 © ISO 2012 – All rights reserved
---------------------- Page: 16 ----------------------
SIST EN ISO 11132:2017
ISO 11132:2012(E)
number of key attributes giving significant interaction, the less consistently the panel is performing. The panel
should receive further training on key attributes that are giving significant interaction.
7.3.3 Repeatability of the panel
The repeatability of the panel can be estimated from the repeatability of the individual assessors. This is
inversely related to the error SD, s :
e
sM= S
e 7
See Table 4.
7.3.4 Reproducibility of the panel
To check for reproducibility of the panel, make evaluations of other samples of the same products at different sessions.
The “between-sessions” factor in a three-way ANOVA (samples, assessors, sessions) should not be significant
at a level of 0,05.
The interaction of samples and sessions should not be significant at a level of 0,05. If it were significant it would
indicate that the evaluation of differences between samples was changing from session to session.
The interaction between assessors and sessions should not be significant at a level of 0,05. If it were significant
it would indicate that the biases of individual assessors were varying from session to session.
If the analysis is being used to describe the performance of the panel as a whole, then the factors in the
ANOVA (sessions, samples and assessors) are random factors. The component SDs may be combined to give
a measure of reproducibility:
Reproducibility SD, s :
R
222 22
s =+sss++ss+
R ea sess as××essprodsess
where
e represents error;
a represents assessors;
sess represents sessions;
prod represents products.
Estimates of bias and variation can be tabulated and/or plotted. Plots over time will show if drifts, step changes
or occasional problems have occurred.
Examples of such presentations are cusum analysis (see Annex B) and Shewhart control charts (see Annex C).
7.4 Individual assessor performance from formal performance validation
7.4.1 Discrimination ability of an assessor
Discrimination ability is measured by the proportion of expected key attributes that have been significantly
discriminated. For each attribute, this is indicated by “between samples” variation significant at a level of 0,05
in the ANOVA table (Table 3). The higher the proportion of key attributes significantly discriminated, the better
the assessor is performing. The assessor should receive further training on expected key attributes that are
not significantly discriminated.
© ISO 2012 – All rights reserved 7
---------------------- Page: 17 --------
...
SLOVENSKI STANDARD
oSIST prEN ISO 11132:2016
01-december-2016
6HQ]RULþQDDQDOL]D0HWRGRORJLMD6PHUQLFH]DQDG]RURYDQMHODVWQRVWL
NYDQWLWDWLYQHJDVHQ]RULþQHJDSDQHOD ,62
Sensory analysis - Methodology - Guidelines for monitoring the performance of a
quantitative sensory panel (ISO 11132:2012)
Sensorische Analyse - Methodologie - Leitlinien zur Überprüfung der Leistungsfähigkeit
eines quantitativen sensorischen Panels (ISO 11132:2012)
Analyse sensorielle - Méthodologie - Lignes directrices pour le contrôle de la
performance d'un jury sensoriel quantitatif (ISO 11132:2012)
Ta slovenski standard je istoveten z: prEN ISO 11132
ICS:
67.240 6HQ]RULþQDDQDOL]D Sensory analysis
oSIST prEN ISO 11132:2016 en
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
---------------------- Page: 1 ----------------------
oSIST prEN ISO 11132:2016
---------------------- Page: 2 ----------------------
oSIST prEN ISO 11132:2016
INTERNATIONAL ISO
STANDARD 11132
First edition
2012-11-01
Sensory analysis — Methodology —
Guidelines for monitoring the performance
of a quantitative sensory panel
Analyse sensorielle — Méthodologie — Lignes directrices pour le
contrôle de la performance d’un jury sensoriel quantitatif
Reference number
ISO 11132:2012(E)
©
ISO 2012
---------------------- Page: 3 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO 2012
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO’s
member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO 2012 – All rights reserved
---------------------- Page: 4 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
Contents Page
Foreword .iv
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Principle . 2
5 Experimental conditions . 4
6 Qualification of assessors . 4
7 Procedure . 4
7.1 Monitoring via formal performance validation . 4
7.2 Statistical analysis of data from formal performance validation (a single session) . 5
7.3 Overall panel performance from formal performance validation . 6
7.4 Individual assessor performance from formal performance validation . 7
7.5 Performance issues . 9
7.6 Monitoring via routine product profiling . 9
7.7 Experimental design for study of performance over time . 9
7.8 Statistical analysis of data over time . 9
7.9 Reproducibility between panels .10
7.10 Statistical analysis of complete profiles .10
Annex A (informative) Example of practical application . 11
Annex B (informative) Example of use of cusum analysis .18
Annex C (informative) Example of use of Shewhart chart .21
Bibliography .23
© ISO 2012 – All rights reserved iii
---------------------- Page: 5 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies
(ISO member bodies). The work of preparing International Standards is normally carried out through ISO
technical committees. Each member body interested in a subject for which a technical committee has been
established has the right to be represented on that committee. International organizations, governmental and
non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the International
Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of technical committees is to prepare International Standards. Draft International Standards
adopted by the technical committees are circulated to the member bodies for voting. Publication as an
International Standard requires approval by at least 75 % of the member bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO shall not be held responsible for identifying any or all such patent rights.
ISO 11132 was prepared by Technical Committee ISO/TC 34, Food products, Subcommittee SC 12,
Sensory analysis.
iv © ISO 2012 – All rights reserved
---------------------- Page: 6 ----------------------
oSIST prEN ISO 11132:2016
INTERNATIONAL STANDARD ISO 11132:2012(E)
Sensory analysis — Methodology — Guidelines for monitoring
the performance of a quantitative sensory panel
1 Scope
This International Standard gives guidelines for monitoring and assessing the overall performance of a
quantitative descriptive panel and the performance of each member.
A panel of assessors can be used as an instrument to assess the magnitude of sensory attributes.
Performance is the measure of the ability of a panel or an assessor to make valid attribute assessments across
the products being evaluated. It can be monitored at a given time point or tracked over time. Performance
comprises the ability of a panel to detect, identify, and measure an attribute, use attributes in a similar way to
other panels or assessors, discriminate between stimuli, use a scale properly, repeat their own results, and
reproduce results from other panels or assessors.
The methods specified allow the consistency, repeatability, freedom from bias and ability to discriminate of
panels and assessors to be monitored and assessed. Monitoring and assessment of agreement between panel
members is also covered. Monitoring and assessment can be carried out in one session or over time.
Monitoring performance data enables the panel leader to improve panel and assessor performance, to
identify issues and retraining needs or to identify assessors who are not performing well enough to continue
participating.
The methods specified in this International Standard can be used by the panel leader to appraise continuously
the performance of panels or individual assessors.
This International Standard applies to individuals or panels in training as well as for established panels.
2 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced document
(including any amendments) applies.
ISO 5492, Sensory analysis — Vocabulary
ISO 8586, Sensory analysis — General guidelines for the selection, training and monitoring of selected and
expert assessors
ISO 8589, Sensory analysis — General guidance for the design of test rooms
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO 5492 and the following apply.
3.1
agreement
ability of different panels or assessors to assign similar scores on a given attribute to samples of the same product
3.2
homogeneity
measure of the agreement of responses among individual assessors within a test session, as a panel of
assessors in replicate sessions, or for an individual assessor in replicate sessions
© ISO 2012 – All rights reserved 1
---------------------- Page: 7 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
3.3
assessor bias
tendency of an assessor to give scores which are consistently above or below the true score when that is
known or the panel mean when it is not
3.4
outlier
an assessment that does not conform to the overall pattern of the data or is extremely different from other
assessments of the same or similar products
3.5
panel drift
phenomenon where a panel, over time, changes in sensitivity or becomes susceptible to biases and as a
consequence changes the location on the scale where an attribute is rated for a constant, reference product
3.6
performance
ability of a panel or an assessor to make valid and reliable assessments of stimuli and stimulus attributes
3.7
repeatability
agreement in assessments of equivalent product samples under the same test conditions by the same
assessor or panel
3.8
reproducibility
agreement in assessments of equivalent product samples under different test conditions, with different tasks
or by a different assessor or panel
NOTE Reproducibility may be measured as any of the following:
— the reproducibility of a panel in the short term, measured between two or more sessions separated by several days;
— the reproducibility of a panel in the medium or long term, measured among sessions separated by several months;
— the reproducibility between different panels, in the same laboratory or in different laboratories;
— the reproducibility of assessments by a single assessor of different attributes of a product.
3.9
validation
process of establishing that sensory data correlate with other data on samples of the same product (e.g.
laboratory measurements, consumer perception, results from other panels, consumer complaints) or that a
panel or assessor is able to meet specified performance criteria
3.10
session
occasion on which products are assessed
NOTE In a single session either one or several products may be assessed by one or several assessors. For an
assessor, whether alone or as part of a panel, sessions are separated in time.
3.11
replicate sessions
sessions in which the assessors, the products, the test conditions, and the task are the same
4 Principle
This International Standard is concerned with sensory panels used to assess the magnitude of one or more
sensory attributes in order to make quantitative descriptions or profiles of products. Different methods are
appropriate to the assessment and monitoring of the performance of panels used for difference testing.
2 © ISO 2012 – All rights reserved
---------------------- Page: 8 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
The performance of a quantitative sensory panel may be evaluated by using assessments already available or
from panel sessions conducted specifically for the purpose of obtaining performance data.
This International Standard may be used either for periodic monitoring or for reviewing ongoing profile data.
A dedicated monitoring procedure at periodic intervals is appropriate for accreditation and other purposes.
Figure 1 is a flow chart for this procedure.
To review ongoing profile data generated by a panel, it can be appropriate to use data that originated from quite
different profiling experiments using different product types, product numbers, etc. The procedure is the same
as that shown in Figure 1. However, as there are no predefined differences, it is recommended that attributes
that are significantly discriminated by the panel as a whole for a given profile be used as the key measures
to check the performance of individual panelists. Attributes that result in no significant difference cannot be
reliably used to check consistency since the lack of agreement within and between panelists probably means
that the products are very similar for those characteristics.
a) Monitoring by means of performance validation.
Use a small set of samples (perhaps three or four) for which some attributes are known to be different.
These attributes are then used as the key measures on which to measure performance.
↓
b) Overall panel performance
1) How many of the expected key attributes have been significantly discriminated?
2) How many of the key attributes show an interaction of sample and assessor? This gives an initial
indication of where there is least consistency across the panel (7.3.2)
3) Repeatability of the panel for the key attributes in replicate sessions (7.3.3)?
↓
c) Individual assessor performance
1) Discrimination ability: how many of the expected key attributes have been significantly discriminated?
2) Repeatability: consistency of discrimination for a given attribute and product (7.4.2)?
3) Contributions to interaction: for which attributes do interactions occur?
i) Interaction due to cross-over effects (7.4.4)
ii) Interaction due to different use of the scale (7.4.5)
↓
d) Where performance issues have been identified, either for the panel or for individual assessors,
appropriate training sessions should be planned.
Figure 1 — Flow chart for performance monitoring
In a single session, the following indicators can be determined.
— Bias of an assessor, measured as the difference between the assessor’s mean and a known, ‘true’ value,
or the mean of the panel as an estimate of the ‘true’ value.
— Repeatability of an assessor, inversely related to the standard deviation (SD) of repeat assessments by the
assessor of the same sample, or between replicates of the same product.
— Reproducibility of an assessor, inversely related to the SD of the assessor’s biases across individual products.
© ISO 2012 – All rights reserved 3
---------------------- Page: 9 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
— Discrimination of an assessor, measured as the ability to assign consistently different scores to
different products.
Bias in an assessor may indicate sensory sensitivity that is different from other assessors and/or use of the
response scale in a way that differs from other assessors.
If an assessor appears to give assessments that differ from those of other assessors, review all the results with
a view to determining whether:
a) the assessments are consistent or variable for repeated samples of the same product;
b) the assessments are similar or different for samples of different products;
c) bias occurs with all, or only some, assessment scales.
Analysis of variance (ANOVA) can be used to investigate these questions.
In some cases, bias may indicate an assessor of superior ability whose results are particularly useful. In other
cases, an assessor showing bias may require retraining or removal from the panel.
A single, consistent approach to statistical analysis of the results is described here. However, some attributes
of panel performance can be assessed by more than one descriptive measure. For instance, error mean
square and error SD (its square root) both express variability in the evaluation of a product. The measures used
should be those that are usual in the field of application.
Other relevant measures of agreement between assessors in the use of the scale for an attribute are the
interaction of assessor and product and the coefficient of correlation between an assessor’s scores and the
panel means. An assessor may have no bias, but may be using the scale in a different way. A correlation close
to 1, a regression slope close to 1, and a regression intercept close to 0 indicate good agreement between an
assessor and the rest of the panel.
With a small number of assessments (fewer than six) the correlation coefficient should be interpreted with
caution, as it can be high (up to 0,7), by chance alone.
5 Experimental conditions
The test facilities shall be in accordance with ISO 8589.
6 Qualification of assessors
The panel shall have the level of qualification and experience of selected assessors (ISO 8586) or better.
7 Procedure
7.1 Monitoring via formal performance validation
At each session, the panel of assessors should be presented with a set of samples similar to those the panel
are to assess when evaluating products and for which statistically significant differences between at least one
pair of the samples can be guaranteed for at least eight attributes.
This number is recommended to encourage panel leaders or sensory managers to identify and select validation
samples that show a realistic as well as a statistical measure of a panel’s performance.
These key attributes are used as key measures against which to assess panel performance. The sample
set should include replicates. There shall be the same number of replicates of each sample. The numbers of
assessors, samples, and replicates depends on the products, the sensory attributes assessed and the purpose
of the procedure. For example 2 or 3, replicates of three or four samples might be used. Care should be taken
to limit the number of assessments required so as to avoid sensory fatigue. The attributes of the samples
should be similar to the range of values that the panel assesses when evaluating products.
4 © ISO 2012 – All rights reserved
---------------------- Page: 10 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
A randomized block experimental design has been adopted, in which the assessors are the “blocks”.
If there is expected to be a carry-over effect from one sample to the next, a suitable experimental design is the
Williams Latin square. The basic design uses four assessors and four samples.
Table 1 — Williams Latin square
Order
Assessor
1 2 3 4
1 A B C D
2 B D A C
3 C A D B
4 D C B A
In this design, each assessor samples the four products in a different order and any particular product is
followed by a different one for each assessor, for example A is followed by B for assessor 1, C for assessor 2,
D for assessor 3 and none for assessor 4.
If multiples of four assessors are available, the same design can be repeated for each set of four.
7.2 Statistical analysis of data from formal performance validation (a single session)
Table 2 illustrates one way to tabulate and summarize the results. Some computer software may require a
different organization of the data, for instance with the samples in columns and the assessors in rows.
Table 2 — Results of the assessors
Assessor
Sample 1 2 j n Mean
q
Scores Mean Scores Mean Scores Mean Scores Mean
Y Y
111 1j1
Y Y
112 1j2
Y
Y Y
1
1.j
11. 1.
Y Y
11n 1jn
r r
2
Y Y
i11
ij1
Y
Y
i12 ij2
Y
Y Y
i
ij.
i1. i.
Y Y
in1 ijn
r r
n
p
Y
Y
Mean
..j
...
In this table it is assumed that there are:
n ≡ number of samples (i = 1,2 … n );
p p
n ≡ number of assessors (j = 1,2 … n );
q q
n ≡ number of replicates per sample (k = 1,2 … n ).
r r
Measures of the performance of the panel as a whole and individual assessors, other than bias, require the
data to be analysed by ANOVA.
The details of the basic calculations are not shown in this International Standard, since the analyses are
normally carried out by a computer package.
© ISO 2012 – All rights reserved 5
---------------------- Page: 11 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
Each assessor’s data are analysed by one-way ANOVA (Table 3).
Table 3 — ANOVA for an individual assessor for one attribute
Source of variation Degrees of freedom Sum of squares Mean square F-ratio
Between samples ν = n - 1 S MS = s /ν
1 p 1 1 1 1
F = MS /MS
1 2
Error ν = n (n - 1) S MS = s /ν
2 p r 2 2 2 2
Total ν = n n - 1 S
3 p r 3
n ≡ number of samples
p
n ≡ number of replicates per sample
r
The data for the complete session are analysed by randomized block ANOVA (Table 4).
Table 4 — ANOVA for a complete session for one attribute
Source of variation Degrees of freedom Sum of squares Mean square F-ratio
Between samples ν = n - 1 S MS = s /ν
4 p 4 4 4 4
a
Between assessors ν = n - 1 S MS = s /ν F = MS /MS
5 q 5 5 5 5 5 7
Interaction ν = (n - 1)(n - 1) S MS = s /ν F = MS /MS
6 p q 6 6 6 6 6 7
Error ν = n n (n - 1) S MS = s /ν
7 p q r 7 7 7 7
Total ν = n n n - 1 S
8 p q r 8
n ≡ number of samples
p
n ≡ number of assessors
q
n ≡ number of replicates per sample
r
a
If the interaction is significant, the F-ratio for between assessors is calculated by F = MS /MS with the interaction mean square in
5 6
the denominator.
7.3 Overall panel performance from formal performance validation
7.3.1 Key attribute discrimination
The proportion of key attributes that have been significantly discriminated as expected should be determined.
For each attribute, this is indicated by significant variation between samples at a level of 0,05 in the ANOVA
table for a session (Table 4). The higher the proportion of key attributes significantly discriminated, the better
the panel is performing. The panel should receive further training on key attributes that are not significantly
discriminated as expected.
7.3.2 Homogeneity of the panel
A panel is not homogeneous when any assessors are in disagreement with the rest of the panel.
A panel is not homogeneous if the interaction of sample and assessor in the ANOVA is significant at a level of 0,05.
The degree of homogeneity of the panel is inversely related to the interaction SD, s .
i
MS −MS
67
s =
i
n
r
See Table 4.
The number of key attributes giving significant interaction of sample and assessor should be determined. Refer
to the ANOVA table for each attribute and note those showing interaction at a level of 0,05. The higher the
6 © ISO 2012 – All rights reserved
---------------------- Page: 12 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
number of key attributes giving significant interaction, the less consistently the panel is performing. The panel
should receive further training on key attributes that are giving significant interaction.
7.3.3 Repeatability of the panel
The repeatability of the panel can be estimated from the repeatability of the individual assessors. This is
inversely related to the error SD, s :
e
sM= S
e 7
See Table 4.
7.3.4 Reproducibility of the panel
To check for reproducibility of the panel, make evaluations of other samples of the same products at different sessions.
The “between-sessions” factor in a three-way ANOVA (samples, assessors, sessions) should not be significant
at a level of 0,05.
The interaction of samples and sessions should not be significant at a level of 0,05. If it were significant it would
indicate that the evaluation of differences between samples was changing from session to session.
The interaction between assessors and sessions should not be significant at a level of 0,05. If it were significant
it would indicate that the biases of individual assessors were varying from session to session.
If the analysis is being used to describe the performance of the panel as a whole, then the factors in the
ANOVA (sessions, samples and assessors) are random factors. The component SDs may be combined to give
a measure of reproducibility:
Reproducibility SD, s :
R
222 22
s =+sss++ss+
R ea sess as××essprodsess
where
e represents error;
a represents assessors;
sess represents sessions;
prod represents products.
Estimates of bias and variation can be tabulated and/or plotted. Plots over time will show if drifts, step changes
or occasional problems have occurred.
Examples of such presentations are cusum analysis (see Annex B) and Shewhart control charts (see Annex C).
7.4 Individual assessor performance from formal performance validation
7.4.1 Discrimination ability of an assessor
Discrimination ability is measured by the proportion of expected key attributes that have been significantly
discriminated. For each attribute, this is indicated by “between samples” variation significant at a level of 0,05
in the ANOVA table (Table 3). The higher the proportion of key attributes significantly discriminated, the better
the assessor is performing. The assessor should receive further training on expected key attributes that are
not significantly discriminated.
© ISO 2012 – All rights reserved 7
---------------------- Page: 13 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
7.4.2 Repeatability of an assessor
The repeatability of an assessor is inversely related to the assessor’s error SD, s :
e
sM= S
e2
See Table 3.
7.4.3 Consistency of an assessor
Consistency of an assessor is inversely related to the SD of the bias terms calculated from each sample.
(For assessor j, the bias term for sample i is the difference between the assessor’s mean for the sample and
the panel mean for the sample,) YY− . See Table 2.
ij.i .
Where it is shown that an assessor’s performance lacks consistency, a scatter diagram of the assessor’s scores
against the panel means, along with regression and correlation analysis, shows whether the inconsistency is
random or has a pattern which indicates different use of the scale from the rest of the panel.
7.4.4 Agreement among assessors
A panel is not homogenous when one or more assessors is in disagreement with the rest of the panel.
This may be detected by:
— an assessor having a significant bias (see Annex B);
— an assessor’s residual SD being significantly greater than for the panel as a whole;
— the correlation coefficient between the assessor’s scores and the panel means being very small or negative.
The slope of the regression of the assessor’s scores on the panel means being significantly different from 1
and/or the intercept being statistically significantly different from 0.
Agreement among the assessors is inversely related to the between-assessors SD, s .
a
MS −MS
57
s =
a
nn
qr
if the interaction was not significant (see Table 4) or
MS −MS
56
s =
a
nn
qr
if the interaction was significant. See Table 4.
Disagreement among the assessors should be tested for significance using the “between assessors” F-ratio
and comparing it with tabulated values of F for the relevant degrees of freedom. If it is significant, there is good
evidence that there is a problem of panel consistency that needs to be addressed. Lack of significance does
not, by itself, give reassurance that there is no problem, because it may be obscured by poor repeatability (a
higher than expected error SD, s ).
e
7.4.5 Different use of scale/bias
A significant ANOVA assessor bias may indicate that assessors use the scale in different ways.
In most cases, no “true” value is known and the overall bias for an assessor is taken to be the difference
between that assessor’s mean and the mean for the panel.
8 © ISO 2012 – All rights reserved
---------------------- Page: 14 ----------------------
oSIST prEN ISO 11132:2016
ISO 11132:2012(E)
Bias for assessor j is given by:
YY−
..j ...
[2]
Scales (see ISO 4121 ) may be used by assessors in different ways. In “universal” scale use, the intensity
of each attribute is rated in relation to the assessor’s knowledge of the total sensory variation that can be
experienced for a specific product type. Panels that work on one or only a few product categories more
commonly develop this approach. In “relative” scale use, the frame of reference used by an assessor for rating
intensity is related to the sensory variation shown by the set of products in a given test. This approach is more
likely to be used by panels that work on a wide range of product categories. To help reduce scaling bias, it is
important to ensure that the scaling approach is consistent within a panel.
7.5 Performance issues
7.5.1 General
Performance issues once identified can be listed and training sessions planned accordingly.
7.5.2 Panel
Training sessions can be organized for the panel as a whole for those attributes causing problems.
7.5.3 Individual assessor
For specific issues with individual assessor performance, it may be appropriate to discuss the problem areas
privately on a one-to-one basis first and follow through with full panel training sessions.
7.6 Monitoring via routine product profiling
The procedure is the same as for monitoring via formal performanc
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.