SIST-V ETSI/EG 201 377-1 V1.3.2:2010
(Main)Speech and multimedia Transmission Quality (STQ) - Specification and measurement of speech transmission quality - Part 1: Introduction to objective comparison measurement methods for one-way speech quality across networks
Speech and multimedia Transmission Quality (STQ) - Specification and measurement of speech transmission quality - Part 1: Introduction to objective comparison measurement methods for one-way speech quality across networks
The present document is part 1 of a series of documents on the specification and measurement of mouth-to-ear (also end-to-end) speech transmission quality. Its main objective is to describe objective comparison-based methods and systems for measuring mouth-to-ear speech quality in networks. Apart from this, it gives an overview on other important aspects of mouth-to-ear speech quality. As the need arises, these other aspects will be covered in more detail in subsequent parts of the present document. Although some of the models described in the present document are superseded the description of the models is kept for information. The present document gives an overview of the methods available for measuring one-way speech transmission quality.
Its purpose is to give information and guidance primarily for operators, users, consumer organizations and regulators who wish to measure or compare the speech transmission quality provided by different networks. The need for the present document has been increased by:
the liberalization of voice services, which has introduced alternative competing providers of voice services;
the introduction of new mobile and IP based technologies;
which has increased the range of services and cost/quality options for users.
The present document applies to both fixed and mobile networks with or without terminal equipment connected to the network. It applies only for narrowband (i.e. between 300 Hz and 3 400 Hz) communications. In principle, comparison methods can be used for IP-based (internet protocol-based) networks, but further work is needed on the calibration of the methods for such networks. The present document describes:
methods for measurements of individual impairments or combinations of impairments to be made at acoustic or electrical interfaces;
methods for combining measures of different impairments into a single objective measure;
methods for predicting the subjective effect of impairments that would be perceived by users.
The methods in the present document assume that subjects with normal hearing have been involved in the test. Therefore, the instrumental methods estimate the perceived speech quality of persons with normal hearing. For each method, the guide contains a general description to highlight the main points, and provides references for more detailed information. The present document does not contain detailed specifications of the individual methods. The present document concentrates on one-way speech quality in networks. It gives no guidance on how to evaluate systems that include equipment such as echo cancellers or in which interactive impairments such as talker echo are significant. The perceived quality in such cases depends not only on the one-way performance, but very much on the behaviour of the equipment under duplex conditions; specifically, the influence of double-talk and delay needs to be considered. Although all assessments of overall speech quality are ultimately subjective because they depend on the user's opinion, a distinction is made between:
subjective methods, which involve real time user assessment; and
objective methods, which use stored information on the user's assessment and therefore involve some degree of calibration. Objective methods for the evaluation of speech quality fall into three categories:
a) Comparison Methods: Methods based on the comparison of transmitted speech signal and a known reference.
b) Absolute Estimation Methods: Methods based on the absolute estimation of the speech quality (i.e. there is no known reference signal); e.g. INMD (ITU-T Recommendation P.561.
c) Transmission Rating Models: Methods that derive a value for the expected speech quality from knowledge about the network; e.g. ETSI Model ETR 250, ITU-T Recommendation G.107.
Kakovost prenosa govora in večpredstavnih vsebin (STQ) - Specifikacija in meritve kakovosti prenosa govora - 1. del: Uvod k objektivnim primerjalnim merilnim metodam za kakovost enosmernih govornih komunikacij
Pričujoči dokument je 1. del serije dokumentov o specifikaciji in meritvah kakovosti prenosa govora med usti in ušesi (med dvema točkama). Njegov glavni namen je opisati objektivne primerjalne metode in sisteme za meritve kakovosti govora med usti in ušesi v omrežjih. Poleg tega podaja pregled drugih pomembnih vidikov kakovosti govora med usti in ušesi. Po potrebi bodo drugi vidiki podrobneje zajeti v naslednjih delih pričujočega dokumenta. Čeprav so nekateri modeli, opisani v tem dokumentu, nadomeščeni, opis modelov ostaja v informativne namene. Pričujoči dokument podaja pregled metod, ki so na voljo za merjenje kakovosti enosmernega govornega prenosa.
Njegov namen je podati informacije in vodilo zlasti za operaterje, uporabnike, potrošniške organizacije in zakonodajalce, ki želijo meriti ali primerjati kakovost prenosa govora, ki jo omogočajo različna omrežja. Potreba po pričujočem dokumentu se je še povečala zaradi:
liberalizacije govornih storitev, ki je pripeljala do drugih konkurenčnih ponudnikov govornih storitev;
uvedbe novih mobilnih tehnologij in tehnologij na osnovi IP;
povečanega obsega storitev in možnosti izbire za uporabnike med stroški in kakovostjo.
Pričujoči dokument velja za fiksna in mobilna omrežja s terminalno opremo, priključeno v omrežje, ali brez nje. Velja le za ozkopasovne (tj. med 300 Hz in 3.400 Hz) komunikacije. Načeloma se primerjalne metode lahko uporabljajo za omrežja na osnovi IP (internetnega protokola), vendar je za kalibracijo metod za takšna omrežja potrebno nadaljnje delo. Pričujoči dokument opisuje:
metode za meritve posameznih okvar ali kombinacije okvar, opravljene na akustičnih ali električnih vmesnikih;
metode za združevanje meril za različne okvare v ukrep z enotnim ciljem;
metode za napovedovanje subjektivnega učinka okvare, kot ga zazna uporabnik.
Metode v pričujočem dokumentu predvidevajo, da so v preskus vključeni preiskovanci z normalnim sluhom. Instrumentalne metode torej ocenjujejo kakovost govora, kot jo zaznajo osebe z normalnim sluhom. Vodilo za vsako metodo vsebuje splošni opis, ki izpostavlja glavne točke, in zagotavlja referenco za podrobnejše informacije. Pričujoči dokument ne vsebuje podrobnejših specifikacij posameznih metod. Pričujoči dokument se osredotoča na kakovost enosmernega govora v omrežjih. Podaja vodilo za način vrednotenja sistemov, ki vključujejo opremo, kot so izločevalniki odmeva, ali pri katerih so interaktivne okvare, kot je odmev govorca, precejšnje. Zaznana kakovost je v teh primerih odvisna ne le od enosmerne zmogljivosti, pač pa v precejšnji meri tudi od obnašanja opreme pod dupleksnimi pogoji; natančneje, treba je upoštevati vpliv dvojnega govora in zakasnitve. V končni fazi so vse ocene skupne kakovosti govora subjektivne, saj so odvisne od uporabnikovega mnenja, razlikujemo pa med:
subjektivnimi metodami, ki vključujejo uporabnikovo oceno v realnem času, in
objektivnimi metodami, ki uporabljajo shranjene informacije o uporabnikovi oceni in torej vključujejo določeno stopnjo kalibracije. Objektivne metode za vrednotenje kakovosti govora so razvrščene v tri kategorije:
a) Primerjalne metode: metode na osnovi primerjave med prenesenim govornim signalom in znano referenco.
b) Metode absolutnega ocenjevanja: metode na osnovi absolutnega ocenjevanje kakovosti govora (tj. brez znanega referenčnega signala); npr. INMD (Priporočilo ITU-T P.561).
c) Modeli za oceno prenosa: metode, ki izpeljujejo vrednost za pričakovano kakovost govora iz poznavanja omrežja; npr. model ETSI ETR 250, Priporočilo ITU-T G.107.
General Information
Buy Standard
Standards Content (Sample)
ETSI EG 201 377-1 V1.3.2 (2009-10)
ETSI Guide
Speech and multimedia Transmission Quality (STQ);
Specification and measurement of
speech transmission quality;
Part 1: Introduction to objective comparison measurement
methods for one-way speech quality across networks
---------------------- Page: 1 ----------------------
2 ETSI EG 201 377-1 V1.3.2 (2009-10)
Reference
REG/STQ-00105-1
Keywords
interworking, quality, speech, testing,
transmission, voice
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2009.
All rights reserved.
TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
---------------------- Page: 2 ----------------------
3 ETSI EG 201 377-1 V1.3.2 (2009-10)
Contents
Intellectual Property Rights . 5
Foreword . 5
1 Scope . 6
2 References . 8
2.1 Normative references . 8
2.2 Informative references . 8
3 Definitions and abbreviations . 11
3.1 Definitions . 11
3.2 Abbreviations . 11
4 Overview . . 12
4.1 Objective . 13
4.2 Related work in standardization . 13
5 Definition of mouth-to-ear speech quality . 14
5.1 General definition . 14
5.2 Human perception characteristics of speech quality . 14
5.2.1 Physical characteristics and psychological impacts . 14
5.2.2 Inter-subject differences . 16
5.2.3 Intra-subject differences . 16
5.2.4 Language-dependent differences . 16
5.3 Network-related issues . 16
5.3.1 Reference configuration for mouth-to-ear measurement . 16
5.3.2 Standardization of quality parameters. 17
5.3.3 Modelling of networks - anomalies . 17
5.4 Terminal equipment related issues . 18
5.5 Technical basis for measurement . 18
5.5.1 Quantification and measurement of speech quality . 18
5.5.2 Required characteristics of speech samples . 19
6 Subjective measurement of speech quality. 20
6.1 Subjective measurement methods . 20
6.2 Application of statistical methods . 20
7 Objective measurement methods . 21
7.1 Basics of speech sample based objective measurement methods . 22
7.2 Pre-processing . 24
7.2.1 Adjustment unit . 24
7.2.2 Modelling and/or measuring transmitter and receiver environment . 24
7.3 Psycho-acoustic sound perception . 24
7.3.1 Time-frequency mapping . 25
7.3.2 Linear prediction coefficients . 25
7.3.3 Cepstrum . 25
7.3.4 Mapping to perceptual (critical band) domain . 26
7.3.5 Frequency masking . 26
7.3.6 Time masking . 27
7.3.7 Psycho-acoustic loudness . 27
7.3.8 Hair cell firing . 28
7.3.9 Specific modelling of annoying components "Relative Approach" . 28
7.4 Comparison of reference and transmitted signal . 29
7.4.1 Euclidean distance . 29
7.4.2 Generalized distance . 29
7.4.3 Asymmetric differences . 30
7.4.4 Distance between probability functions . 30
7.4.5 Multi-resolution analysis . 30
7.4.6 Compression to single number. 30
ETSI
---------------------- Page: 3 ----------------------
4 ETSI EG 201 377-1 V1.3.2 (2009-10)
7.4.7 Mapping to MOS scale . 30
7.5 Comparability of Objective Model Results . 31
7.5.1 Comparison of Results between Models . 31
7.5.2 Comparison of Results of one Model Implemented in Different Test Equipment . 31
7.5.3 Comparison of Results of one Model in Different Scenarios . 31
7.5.4 Optimization of Systems based on the Results of One Model . 31
8 Overview of INMD . 32
9 Overview of Single-ended Objective Speech Quality Assessment . 32
10 Overview of the E-Model . . 33
11 Use of building blocks in some known systems . 34
11.1 Comparison-based schemes. 34
11.2 E-Model . 35
Annex A: Examples of specific systems . 36
A.1 Perceptual Speech Quality Measure (PSQM) . 36
A.2 Measuring Normalizing Blocks (MNB) . 37
A.3 PACE . 38
A.4 Telecommunication Objective Speech Quality Assessment (TOSQA) . 39
A.5 Perceptual Analysis/Measurement System (PAMS) . 40
A.6 ITU-T Recommendation P.862: Perceptual Evaluation of Speech Quality . 41
A.7 EG 202 396-3: Background noise transmission - Objective test methods . 42
Annex B: Terminal equipment related issues . 44
B.1 Overview . . 44
Annex C: Subjective measurement methods . 47
C.1 Absolute Category Rating (ACR) . 47
C.2 Degradation Category Rating (DCR) . 47
C.3 Comparison Category Rating (CCR) . 47
C.4 Interview and survey test . 48
C.5 Conversational tests . 48
C.6 Double talk tests . 49
C.7 Talking and listening tests . 49
C.8 Listening-only test procedure . 49
Annex D: Application of statistical methods . 50
D.1 Statistical relevance of results . 50
D.2 Estimation of confidence intervals . 51
D.3 ANOVA . . 52
Annex E: Bibliography . 53
History . 54
ETSI
---------------------- Page: 4 ----------------------
5 ETSI EG 201 377-1 V1.3.2 (2009-10)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality
(STQ).
The present document is part 1 of a multi-part deliverable covering the specification and measurement of speech
transmission quality, as identified below:
Part 1: "Introduction to objective comparison measurement methods for one-way speech quality across
networks";
Part 2: "Mouth-to-ear speech transmission quality including terminals";
Part 3: "Non-intrusive objective measurement methods applicable to networks and links with classes of
services".
ETSI
---------------------- Page: 5 ----------------------
6 ETSI EG 201 377-1 V1.3.2 (2009-10)
1 Scope
The present document is part 1 of a series of documents on the specification and measurement of mouth-to-ear (also
end-to-end) speech transmission quality. Its main objective is to describe objective comparison-based methods and
systems for measuring mouth-to-ear speech quality in networks. Apart from this, it gives an overview on other
important aspects of mouth-to-ear speech quality. As the need arises, these other aspects will be covered in more detail
in subsequent parts of the present document. Although some of the models described in the present document are
superseded the description of the models is kept for information.
The present document gives an overview of the methods available for measuring one-way speech transmission quality.
Its purpose is to give information and guidance primarily for operators, users, consumer organizations and regulators
who wish to measure or compare the speech transmission quality provided by different networks. The need for the
present document has been increased by:
• the liberalization of voice services, which has introduced alternative competing providers of voice services;
• the introduction of new mobile and IP based technologies;
which has increased the range of services and cost/quality options for users.
The present document applies to both fixed and mobile networks with or without terminal equipment connected to the
network. It applies only for narrowband (i.e. between 300 Hz and 3 400 Hz) communications. In principle, comparison
methods can be used for IP-based (internet protocol-based) networks, but further work is needed on the calibration of
the methods for such networks. The present document describes:
• methods for measurements of individual impairments or combinations of impairments to be made at acoustic
or electrical interfaces;
• methods for combining measures of different impairments into a single objective measure;
• methods for predicting the subjective effect of impairments that would be perceived by users.
The methods in the present document assume that subjects with normal hearing have been involved in the test.
Therefore, the instrumental methods estimate the perceived speech quality of persons with normal hearing. For each
method, the guide contains a general description to highlight the main points, and provides references for more detailed
information. The present document does not contain detailed specifications of the individual methods.
The present document concentrates on one-way speech quality in networks. It gives no guidance on how to evaluate
systems that include equipment such as echo cancellers or in which interactive impairments such as talker echo are
significant. The perceived quality in such cases depends not only on the one-way performance, but very much on the
behaviour of the equipment under duplex conditions; specifically, the influence of double-talk and delay needs to be
considered.
Although all assessments of overall speech quality are ultimately subjective because they depend on the user's opinion,
a distinction is made between:
• subjective methods, which involve real time user assessment; and
• objective methods, which use stored information on the user's assessment and therefore involve some degree
of calibration.
Objective methods for the evaluation of speech quality fall into three categories:
a) Comparison Methods: Methods based on the comparison of transmitted speech signal and a known reference.
b) Absolute Estimation Methods: Methods based on the absolute estimation of the speech quality (i.e. there is no
known reference signal); e.g. INMD (ITU-T Recommendation P.561 [i.16]).
c) Transmission Rating Models: Methods that derive a value for the expected speech quality from knowledge
about the network; e.g. ETSI Model (ETR 250 [i.1], ITU-T Recommendation G.107 [i.14]).
ETSI
---------------------- Page: 6 ----------------------
7 ETSI EG 201 377-1 V1.3.2 (2009-10)
The classification of assessment methods is depicted in figure 1.
Practical implementations of test equipment may include combinations of these methods. The focus of the present
document is on comparison methods (intrusive methods), which currently yield the most accurate results. The other
categories are only covered in short overviews, although they may be preferable for certain applications.
Subjective and Objective Methods
Users' subjective
assessment
Real - time Stored experience
assessment (i.e., past assessments)
Additional
experience
Subjective methods
Objective methods
Calibration
e.g., listening- only and conversational
Additional information
e.g.: radio link budget
error rate
cell loss rate
Signal - based measurements
Parameter- based models
e.g. speech samples and other test signals
Transmission Rating
Comparison Absolute Estimation
Models
Measure combined effect Predict combined effect of E - Model
of all impairments individually measured impairments
Talker Talker
Measure-
Parameters of:
ment Computa-
Network Network
"Talker",
Measure-
tion using
Network,
ment
knowledge
"Listener"
of network
Listener Listener
a) b) c)
Figure 1: Classification of assessment methods showing:
a) Comparison methods,
b) Absolute estimation methods,
c) Transmission rating models
NOTE: As an ETSI Guide, the present document provides guidelines for test methods that may be implemented.
However, a test method and especially quality models can only be applied in the way and within the
scope defined in the reference standard. A "Warning" indicates when this applies.
ETSI
---------------------- Page: 7 ----------------------
8 ETSI EG 201 377-1 V1.3.2 (2009-10)
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific.
• For a specific reference, subsequent revisions do not apply.
• Non-specific reference may be made only to a complete document or a part thereof and only in the following
cases:
- if it is accepted that it will be possible to use all future changes of the referenced document for the
purposes of the referring document;
- for informative references.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are indispensable for the application of the present document. For dated
references, only the edition cited applies. For non-specific references, the latest edition of the referenced document
(including any amendments) applies.
Not applicable.
2.2 Informative references
The following referenced documents are not essential to the use of the present document but they assist the user with
regard to a particular subject area. For non-specific references, the latest version of the referenced document (including
any amendments) applies.
[i.1] ETSI ETR 250: "Transmission and Multiplexing (TM); Speech communication quality from
mouth to ear for 3,1 kHz handset telephony across networks".
[i.2] ETSI EG 201 050: "Speech Processing, Transmission and Quality Aspects (STQ); Overall
Transmission Plan Aspects for Telephony in a Private Network".
[i.3] ETSI TR 102 082: "Speech Processing, Transmission and Quality Aspects (STQ); Guidance on
writing specifications and tests for non-linear and time variant telephony terminals".
[i.4] ETSI EG 202 396-1: "Speech and multimedia Transmission Quality (STQ); Speech quality
performance in the presence of background noise; Part 1: Background noise simulation technique
and background noise database".
[i.5] ETSI EG 202 396-2: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
quality performance in the presence of background noise; Part 2: Background noise transmission -
Network simulation - Subjective test database and results".
[i.6] ETSI EG 202 396-3: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
quality performance in the presence of background noise; Part 3: Background noise transmission -
Objective test methods".
[i.7] ETSI ES 202 737: "Speech and multimedia Transmission Quality (STQ); Transmission
requirements for narrowband VoIP terminals (handset and headset) from a QoS perspective as
perceived by the user".
ETSI
---------------------- Page: 8 ----------------------
9 ETSI EG 201 377-1 V1.3.2 (2009-10)
[i.8] ETSI ES 202 738: "Speech and multimedia Transmission Quality (STQ); Transmission
requirements for narrowband VoIP loudspeaking and handsfree terminals from a QoS perspective
as perceived by the user".
[i.9] ETSI ES 202 739: "Speech and multimediaTransmission Quality (STQ); Transmission
requirements for wideband VoIP terminals (handset and headset) from a QoS perspective as
perceived by the user".
[i.10] ETSI ES 202 740: "Speech and multimedia Transmission Quality (STQ); Transmission
requirements for wideband VoIP loudspeaking and handsfree terminals from a QoS perspective as
perceived by the user".
[i.11] EURESCOM Project P603 vol.1: "Quality of Service: Measurement Method Selection;
Deliverable 2: Measurement Method; Volume 1 of 2: Main Report".
[i.12] EURESCOM Project P603 vol.2: "Quality of Service: Measurement Method Selection;
Deliverable 2: Measurement Method; Volume 2 of 2: Annexes".
[i.13] ISO 532 (1975): "Acoustics - Method for calculating loudness level".
[i.14] ITU-T Recommendation G.107 (2002): "The E-model, a computational model for use in
transmission planning".
[i.15] ITU-T Recommendation P.501: "Test signals for use in telephonometry".
[i.16] ITU-T Recommendation P.561 (2002): "In-service, non-intrusive measurement device - voice
service measurements".
[i.17] ITU-T Recommendation P.562 (2004): "Analysis and interpretation of INMD voice-service
measurements".
[i.18] ITU-T Recommendation P.563 (2004): "Single-ended method for objective speech quality
assessment in narrow-band telephony applications".
[i.19] ITU-T Recommendation P.800 (1996): "Methods for subjective determination of transmission
quality".
[i.20] ITU-T Recommendation P.830 (1996): "Subjective performance assessment of telephone-band
and wideband digital codecs".
[i.21] ITU-T Recommendation P.835 (2003): " Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[i.22] ITU-T COM12-20: "Improvement of the P.861 perceptual speech quality measure".
[i.23] ITU-T COM12-24: "Proposed Annex A to Recommendation P.861".
[i.24] ITU-T COM12-34: "TOSQA - Telecommunication objective speech quality assessment".
[i.25] ITU-T COM12-62: "Results of Processing ITU speech database supplement 23 with the
end-to-end quality assessment algorithm "PACE"".
[i.26] Journal of the Audio Engineering Society: "A Perceptual Audio Quality Measure Based on a
Psychoacoustic Sound Representation", Beerends J.G., Stemerdink J.A. (1992), vol. 40, no. 12,
pp. 963-978.
[i.27] BT Engineering Journal (1998): "Getting the message loud and clear: quantifying call clarity",
Broom, S.; Coackley, P.; Sheppard, P., Vol. 17, p. 66-72.
[i.28] Speech Communication (1994): "Auditory Distortion Measure for Speech Coder Evaluation -
Discrimination Information Approach", De A., Kabal P., 14(3):205-229.
[i.29] McMillan Publishing Company (1993): "Discrete Time Processing of Speech Signals", Deller J.R.,
Proakis J.G., Hansen J.H.L. Eaglewood Cliffs NJ.
ETSI
---------------------- Page: 9 ----------------------
10 ETSI EG 201 377-1 V1.3.2 (2009-10)
[i.30] Report TA No. 92, KTH Karolinska Institutet(1979): "Statistical treatment of data from listening
tests on sound-reproducing systems", Gabrielsson A, Department of Technical Audiology,
S-10044 Stockholm, Sweden.
[i.31] Prentice Hall Press (1995): "Introduction to Mathematical Statistics", Hogg R.V., Craig A.T.,
Eaglewood Cliffs.
[i.32] IEEE Proc
...
Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
ETSI Guide
Speech and multimedia Transmission Quality (STQ);
Specification and measurement of
speech transmission quality;
Part 1: Introduction to objective comparison measurement
methods for one-way speech quality across networks
---------------------- Page: 1 ----------------------
2 Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
Reference
REG/STQ-00105-1
Keywords
interworking, quality, speech, testing,
transmission, voice
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2009.
All rights reserved.
TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
---------------------- Page: 2 ----------------------
3 Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
Contents
Intellectual Property Rights . 5
Foreword . 5
1 Scope . 6
2 References . 8
2.1 Normative references . 8
2.2 Informative references . 8
3 Definitions and abbreviations . 11
3.1 Definitions . 11
3.2 Abbreviations . 11
4 Overview . . 12
4.1 Objective . 13
4.2 Related work in standardization . 13
5 Definition of mouth-to-ear speech quality . 14
5.1 General definition . 14
5.2 Human perception characteristics of speech quality . 14
5.2.1 Physical characteristics and psychological impacts . 14
5.2.2 Inter-subject differences . 16
5.2.3 Intra-subject differences . 16
5.2.4 Language-dependent differences . 16
5.3 Network-related issues . 16
5.3.1 Reference configuration for mouth-to-ear measurement . 16
5.3.2 Standardization of quality parameters. 17
5.3.3 Modelling of networks - anomalies . 17
5.4 Terminal equipment related issues . 18
5.5 Technical basis for measurement . 18
5.5.1 Quantification and measurement of speech quality . 18
5.5.2 Required characteristics of speech samples . 19
6 Subjective measurement of speech quality. 20
6.1 Subjective measurement methods . 20
6.2 Application of statistical methods . 20
7 Objective measurement methods . 21
7.1 Basics of speech sample based objective measurement methods . 22
7.2 Pre-processing . 24
7.2.1 Adjustment unit . 24
7.2.2 Modelling and/or measuring transmitter and receiver environment . 24
7.3 Psycho-acoustic sound perception . 24
7.3.1 Time-frequency mapping . 25
7.3.2 Linear prediction coefficients . 25
7.3.3 Cepstrum . 25
7.3.4 Mapping to perceptual (critical band) domain . 26
7.3.5 Frequency masking . 26
7.3.6 Time masking . 27
7.3.7 Psycho-acoustic loudness . 27
7.3.8 Hair cell firing . 28
7.3.9 Specific modelling of annoying components "Relative Approach" . 28
7.4 Comparison of reference and transmitted signal . 29
7.4.1 Euclidean distance . 29
7.4.2 Generalized distance . 29
7.4.3 Asymmetric differences . 30
7.4.4 Distance between probability functions . 30
7.4.5 Multi-resolution analysis . 30
7.4.6 Compression to single number. 30
ETSI
---------------------- Page: 3 ----------------------
4 Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
7.4.7 Mapping to MOS scale . 30
7.5 Comparability of Objective Model Results . 31
7.5.1 Comparison of Results between Models . 31
7.5.2 Comparison of Results of one Model Implemented in Different Test Equipment . 31
7.5.3 Comparison of Results of one Model in Different Scenarios . 31
7.5.4 Optimization of Systems based on the Results of One Model . 31
8 Overview of INMD . 32
9 Overview of Single-ended Objective Speech Quality Assessment . 32
10 Overview of the E-Model . . 33
11 Use of building blocks in some known systems . 34
11.1 Comparison-based schemes. 34
11.2 E-Model . 35
Annex A: Examples of specific systems . 36
A.1 Perceptual Speech Quality Measure (PSQM) . 36
A.2 Measuring Normalizing Blocks (MNB) . 37
A.3 PACE . 38
A.4 Telecommunication Objective Speech Quality Assessment (TOSQA) . 39
A.5 Perceptual Analysis/Measurement System (PAMS) . 40
A.6 ITU-T Recommendation P.862: Perceptual Evaluation of Speech Quality . 41
A.7 EG 202 396-3: Background noise transmission - Objective test methods . 42
Annex B: Terminal equipment related issues . 44
B.1 Overview . . 44
Annex C: Subjective measurement methods . 47
C.1 Absolute Category Rating (ACR) . 47
C.2 Degradation Category Rating (DCR) . 47
C.3 Comparison Category Rating (CCR) . 47
C.4 Interview and survey test . 48
C.5 Conversational tests . 48
C.6 Double talk tests . 49
C.7 Talking and listening tests . 49
C.8 Listening-only test procedure . 49
Annex D: Application of statistical methods . 50
D.1 Statistical relevance of results . 50
D.2 Estimation of confidence intervals . 51
D.3 ANOVA . . 52
Annex E: Bibliography . 53
History . 54
ETSI
---------------------- Page: 4 ----------------------
5 Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality
(STQ), and is now submitted for the ETSI standards Membership Approval Procedure.
The present document is part 1 of a multi-part deliverable covering the specification and measurement of speech
transmission quality, as identified below:
Part 1: "Introduction to objective comparison measurement methods for one-way speech quality across
networks";
Part 2: "Mouth-to-ear speech transmission quality including terminals";
Part 3: "Non-intrusive objective measurement methods applicable to networks and links with classes of
services".
ETSI
---------------------- Page: 5 ----------------------
6 Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
1 Scope
The present document is part 1 of a series of documents on the specification and measurement of mouth-to-ear (also
end-to-end) speech transmission quality. Its main objective is to describe objective comparison-based methods and
systems for measuring mouth-to-ear speech quality in networks. Apart from this, it gives an overview on other
important aspects of mouth-to-ear speech quality. As the need arises, these other aspects will be covered in more detail
in subsequent parts of the present document. Although some of the models described in the present document are
superseded the description of the models is kept for information.
The present document gives an overview of the methods available for measuring one-way speech transmission quality.
Its purpose is to give information and guidance primarily for operators, users, consumer organizations and regulators
who wish to measure or compare the speech transmission quality provided by different networks. The need for the
present document has been increased by:
• the liberalization of voice services, which has introduced alternative competing providers of voice services;
• the introduction of new mobile and IP based technologies;
which has increased the range of services and cost/quality options for users.
The present document applies to both fixed and mobile networks with or without terminal equipment connected to the
network. It applies only for narrowband (i.e. between 300 Hz and 3 400 Hz) communications. In principle, comparison
methods can be used for IP-based (internet protocol-based) networks, but further work is needed on the calibration of
the methods for such networks. The present document describes:
• methods for measurements of individual impairments or combinations of impairments to be made at acoustic
or electrical interfaces;
• methods for combining measures of different impairments into a single objective measure;
• methods for predicting the subjective effect of impairments that would be perceived by users.
The methods in the present document assume that subjects with normal hearing have been involved in the test.
Therefore, the instrumental methods estimate the perceived speech quality of persons with normal hearing. For each
method, the guide contains a general description to highlight the main points, and provides references for more detailed
information. The present document does not contain detailed specifications of the individual methods.
The present document concentrates on one-way speech quality in networks. It gives no guidance on how to evaluate
systems that include equipment such as echo cancellers or in which interactive impairments such as talker echo are
significant. The perceived quality in such cases depends not only on the one-way performance, but very much on the
behaviour of the equipment under duplex conditions; specifically, the influence of double-talk and delay needs to be
considered.
Although all assessments of overall speech quality are ultimately subjective because they depend on the user's opinion,
a distinction is made between:
• subjective methods, which involve real time user assessment; and
• objective methods, which use stored information on the user's assessment and therefore involve some degree
of calibration.
Objective methods for the evaluation of speech quality fall into three categories:
a) Comparison Methods: Methods based on the comparison of transmitted speech signal and a known reference.
b) Absolute Estimation Methods: Methods based on the absolute estimation of the speech quality (i.e. there is no
known reference signal); e.g. INMD (ITU-T Recommendation P.561 [i.16]).
c) Transmission Rating Models: Methods that derive a value for the expected speech quality from knowledge
about the network; e.g. ETSI Model (ETR 250 [i.1], ITU-T Recommendation G.107 [i.14]).
ETSI
---------------------- Page: 6 ----------------------
7 Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
The classification of assessment methods is depicted in figure 1.
Practical implementations of test equipment may include combinations of these methods. The focus of the present
document is on comparison methods (intrusive methods), which currently yield the most accurate results. The other
categories are only covered in short overviews, although they may be preferable for certain applications.
Subjective and Objective Methods
Users' subjective
assessment
Real - time Stored experience
assessment (i.e., past assessments)
Additional
experience
Subjective methods
Objective methods
Calibration
e.g., listening- only and conversational
Additional information
e.g.: radio link budget
error rate
cell loss rate
Signal - based measurements
Parameter- based models
e.g. speech samples and other test signals
Transmission Rating
Comparison Absolute Estimation
Models
Measure combined effect Predict combined effect of E - Model
of all impairments individually measured impairments
Talker Talker
Measure-
Parameters of:
ment Computa-
Network Network
"Talker",
Measure-
tion using
Network,
ment
knowledge
"Listener"
of network
Listener Listener
a) b) c)
Figure 1: Classification of assessment methods showing:
a) Comparison methods,
b) Absolute estimation methods,
c) Transmission rating models
NOTE: As an ETSI Guide, the present document provides guidelines for test methods that may be implemented.
However, a test method and especially quality models can only be applied in the way and within the
scope defined in the reference standard. A "Warning" indicates when this applies.
ETSI
---------------------- Page: 7 ----------------------
8 Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific.
• For a specific reference, subsequent revisions do not apply.
• Non-specific reference may be made only to a complete document or a part thereof and only in the following
cases:
- if it is accepted that it will be possible to use all future changes of the referenced document for the
purposes of the referring document;
- for informative references.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are indispensable for the application of the present document. For dated
references, only the edition cited applies. For non-specific references, the latest edition of the referenced document
(including any amendments) applies.
Not applicable.
2.2 Informative references
The following referenced documents are not essential to the use of the present document but they assist the user with
regard to a particular subject area. For non-specific references, the latest version of the referenced document (including
any amendments) applies.
[i.1] ETSI ETR 250: "Transmission and Multiplexing (TM); Speech communication quality from
mouth to ear for 3,1 kHz handset telephony across networks".
[i.2] ETSI EG 201 050: "Speech Processing, Transmission and Quality Aspects (STQ); Overall
Transmission Plan Aspects for Telephony in a Private Network".
[i.3] ETSI TR 102 082: "Speech Processing, Transmission and Quality Aspects (STQ); Guidance on
writing specifications and tests for non-linear and time variant telephony terminals".
[i.4] ETSI EG 202 396-1: "Speech and multimedia Transmission Quality (STQ); Speech quality
performance in the presence of background noise; Part 1: Background noise simulation technique
and background noise database".
[i.5] ETSI EG 202 396-2: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
quality performance in the presence of background noise; Part 2: Background noise transmission -
Network simulation - Subjective test database and results".
[i.6] ETSI EG 202 396-3: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
quality performance in the presence of background noise; Part 3: Background noise transmission -
Objective test methods".
[i.7] ETSI ES 202 737: "Speech and multimedia Transmission Quality (STQ); Transmission
requirements for narrowband VoIP terminals (handset and headset) from a QoS perspective as
perceived by the user".
ETSI
---------------------- Page: 8 ----------------------
9 Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
[i.8] ETSI ES 202 738: "Speech and multimedia Transmission Quality (STQ); Transmission
requirements for narrowband VoIP loudspeaking and handsfree terminals from a QoS perspective
as perceived by the user".
[i.9] ETSI ES 202 739: "Speech and multimediaTransmission Quality (STQ); Transmission
requirements for wideband VoIP terminals (handset and headset) from a QoS perspective as
perceived by the user".
[i.10] ETSI ES 202 740: "Speech and multimedia Transmission Quality (STQ); Transmission
requirements for wideband VoIP loudspeaking and handsfree terminals from a QoS perspective as
perceived by the user".
[i.11] EURESCOM Project P603 vol.1: "Quality of Service: Measurement Method Selection;
Deliverable 2: Measurement Method; Volume 1 of 2: Main Report".
[i.12] EURESCOM Project P603 vol.2: "Quality of Service: Measurement Method Selection;
Deliverable 2: Measurement Method; Volume 2 of 2: Annexes".
[i.13] ISO 532 (1975): "Acoustics - Method for calculating loudness level".
[i.14] ITU-T Recommendation G.107 (2002): "The E-model, a computational model for use in
transmission planning".
[i.15] ITU-T Recommendation P.501: "Test signals for use in telephonometry".
[i.16] ITU-T Recommendation P.561 (2002): "In-service, non-intrusive measurement device - voice
service measurements".
[i.17] ITU-T Recommendation P.562 (2004): "Analysis and interpretation of INMD voice-service
measurements".
[i.18] ITU-T Recommendation P.563 (2004): "Single-ended method for objective speech quality
assessment in narrow-band telephony applications".
[i.19] ITU-T Recommendation P.800 (1996): "Methods for subjective determination of transmission
quality".
[i.20] ITU-T Recommendation P.830 (1996): "Subjective performance assessment of telephone-band
and wideband digital codecs".
[i.21] ITU-T Recommendation P.835 (2003): " Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[i.22] ITU-T COM12-20: "Improvement of the P.861 perceptual speech quality measure".
[i.23] ITU-T COM12-24: "Proposed Annex A to Recommendation P.861".
[i.24] ITU-T COM12-34: "TOSQA - Telecommunication objective speech quality assessment".
[i.25] ITU-T COM12-62: "Results of Processing ITU speech database supplement 23 with the
end-to-end quality assessment algorithm "PACE"".
[i.26] Journal of the Audio Engineering Society: "A Perceptual Audio Quality Measure Based on a
Psychoacoustic Sound Representation", Beerends J.G., Stemerdink J.A. (1992), vol. 40, no. 12,
pp. 963-978.
[i.27] BT Engineering Journal (1998): "Getting the message loud and clear: quantifying call clarity",
Broom, S.; Coackley, P.; Sheppard, P., Vol. 17, p. 66-72.
[i.28] Speech Communication (1994): "Auditory Distortion Measure for Speech Coder Evaluation -
Discrimination Information Approach", De A., Kabal P., 14(3):205-229.
[i.29] McMillan Publishing Company (1993): "Discrete Time Processing of Speech Signals", Deller J.R.,
Proakis J.G., Hansen J.H.L. Eaglewood Cliffs NJ.
ETSI
---------------------- Page: 9 ----------------------
10 Final draft ETSI EG 201 377-1 V1.3.2 (2009-08)
[i.30] Report TA No. 92, KTH Karolinska Institutet(1979): "Statistical treatment of data from listening
tests on sound-reproducing systems", Gabrielsson A, Depart
...
SLOVENSKI STANDARD
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
01-februar-2010
.DNRYRVWSUHQRVDJRYRUDLQYHþSUHGVWDYQLKYVHELQ 674 6SHFLILNDFLMDLQ
PHULWYHNDNRYRVWLSUHQRVDJRYRUDGHO 8YRGNREMHNWLYQLPSULPHUMDOQLP
PHULOQLPPHWRGDP]DNDNRYRVWHQRVPHUQLKJRYRUQLKNRPXQLNDFLM
Speech and multimedia Transmission Quality (STQ) - Specification and measurement of
speech transmission quality - Part 1: Introduction to objective comparison measurement
methods for one-way speech quality across networks
Ta slovenski standard je istoveten z: EG 201 377-1 Version 1.3.2
ICS:
33.040.35 Telefonska omrežja Telephone networks
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010 en
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
---------------------- Page: 1 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
---------------------- Page: 2 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
ETSI EG 201 377-1 V1.3.2 (2009-10)
ETSI Guide
Speech and multimedia Transmission Quality (STQ);
Specification and measurement of
speech transmission quality;
Part 1: Introduction to objective comparison measurement
methods for one-way speech quality across networks
---------------------- Page: 3 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
2 ETSI EG 201 377-1 V1.3.2 (2009-10)
Reference
REG/STQ-00105-1
Keywords
interworking, quality, speech, testing,
transmission, voice
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2009.
All rights reserved.
TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
---------------------- Page: 4 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
3 ETSI EG 201 377-1 V1.3.2 (2009-10)
Contents
Intellectual Property Rights . 5
Foreword . 5
1 Scope . 6
2 References . 8
2.1 Normative references . 8
2.2 Informative references . 8
3 Definitions and abbreviations . 11
3.1 Definitions . 11
3.2 Abbreviations . 11
4 Overview . . 12
4.1 Objective . 13
4.2 Related work in standardization . 13
5 Definition of mouth-to-ear speech quality . 14
5.1 General definition . 14
5.2 Human perception characteristics of speech quality . 14
5.2.1 Physical characteristics and psychological impacts . 14
5.2.2 Inter-subject differences . 16
5.2.3 Intra-subject differences . 16
5.2.4 Language-dependent differences . 16
5.3 Network-related issues . 16
5.3.1 Reference configuration for mouth-to-ear measurement . 16
5.3.2 Standardization of quality parameters. 17
5.3.3 Modelling of networks - anomalies . 17
5.4 Terminal equipment related issues . 18
5.5 Technical basis for measurement . 18
5.5.1 Quantification and measurement of speech quality . 18
5.5.2 Required characteristics of speech samples . 19
6 Subjective measurement of speech quality. 20
6.1 Subjective measurement methods . 20
6.2 Application of statistical methods . 20
7 Objective measurement methods . 21
7.1 Basics of speech sample based objective measurement methods . 22
7.2 Pre-processing . 24
7.2.1 Adjustment unit . 24
7.2.2 Modelling and/or measuring transmitter and receiver environment . 24
7.3 Psycho-acoustic sound perception . 24
7.3.1 Time-frequency mapping . 25
7.3.2 Linear prediction coefficients . 25
7.3.3 Cepstrum . 25
7.3.4 Mapping to perceptual (critical band) domain . 26
7.3.5 Frequency masking . 26
7.3.6 Time masking . 27
7.3.7 Psycho-acoustic loudness . 27
7.3.8 Hair cell firing . 28
7.3.9 Specific modelling of annoying components "Relative Approach" . 28
7.4 Comparison of reference and transmitted signal . 29
7.4.1 Euclidean distance . 29
7.4.2 Generalized distance . 29
7.4.3 Asymmetric differences . 30
7.4.4 Distance between probability functions . 30
7.4.5 Multi-resolution analysis . 30
7.4.6 Compression to single number. 30
ETSI
---------------------- Page: 5 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
4 ETSI EG 201 377-1 V1.3.2 (2009-10)
7.4.7 Mapping to MOS scale . 30
7.5 Comparability of Objective Model Results . 31
7.5.1 Comparison of Results between Models . 31
7.5.2 Comparison of Results of one Model Implemented in Different Test Equipment . 31
7.5.3 Comparison of Results of one Model in Different Scenarios . 31
7.5.4 Optimization of Systems based on the Results of One Model . 31
8 Overview of INMD . 32
9 Overview of Single-ended Objective Speech Quality Assessment . 32
10 Overview of the E-Model . . 33
11 Use of building blocks in some known systems . 34
11.1 Comparison-based schemes. 34
11.2 E-Model . 35
Annex A: Examples of specific systems . 36
A.1 Perceptual Speech Quality Measure (PSQM) . 36
A.2 Measuring Normalizing Blocks (MNB) . 37
A.3 PACE . 38
A.4 Telecommunication Objective Speech Quality Assessment (TOSQA) . 39
A.5 Perceptual Analysis/Measurement System (PAMS) . 40
A.6 ITU-T Recommendation P.862: Perceptual Evaluation of Speech Quality . 41
A.7 EG 202 396-3: Background noise transmission - Objective test methods . 42
Annex B: Terminal equipment related issues . 44
B.1 Overview . . 44
Annex C: Subjective measurement methods . 47
C.1 Absolute Category Rating (ACR) . 47
C.2 Degradation Category Rating (DCR) . 47
C.3 Comparison Category Rating (CCR) . 47
C.4 Interview and survey test . 48
C.5 Conversational tests . 48
C.6 Double talk tests . 49
C.7 Talking and listening tests . 49
C.8 Listening-only test procedure . 49
Annex D: Application of statistical methods . 50
D.1 Statistical relevance of results . 50
D.2 Estimation of confidence intervals . 51
D.3 ANOVA . . 52
Annex E: Bibliography . 53
History . 54
ETSI
---------------------- Page: 6 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
5 ETSI EG 201 377-1 V1.3.2 (2009-10)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality
(STQ).
The present document is part 1 of a multi-part deliverable covering the specification and measurement of speech
transmission quality, as identified below:
Part 1: "Introduction to objective comparison measurement methods for one-way speech quality across
networks";
Part 2: "Mouth-to-ear speech transmission quality including terminals";
Part 3: "Non-intrusive objective measurement methods applicable to networks and links with classes of
services".
ETSI
---------------------- Page: 7 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
6 ETSI EG 201 377-1 V1.3.2 (2009-10)
1 Scope
The present document is part 1 of a series of documents on the specification and measurement of mouth-to-ear (also
end-to-end) speech transmission quality. Its main objective is to describe objective comparison-based methods and
systems for measuring mouth-to-ear speech quality in networks. Apart from this, it gives an overview on other
important aspects of mouth-to-ear speech quality. As the need arises, these other aspects will be covered in more detail
in subsequent parts of the present document. Although some of the models described in the present document are
superseded the description of the models is kept for information.
The present document gives an overview of the methods available for measuring one-way speech transmission quality.
Its purpose is to give information and guidance primarily for operators, users, consumer organizations and regulators
who wish to measure or compare the speech transmission quality provided by different networks. The need for the
present document has been increased by:
• the liberalization of voice services, which has introduced alternative competing providers of voice services;
• the introduction of new mobile and IP based technologies;
which has increased the range of services and cost/quality options for users.
The present document applies to both fixed and mobile networks with or without terminal equipment connected to the
network. It applies only for narrowband (i.e. between 300 Hz and 3 400 Hz) communications. In principle, comparison
methods can be used for IP-based (internet protocol-based) networks, but further work is needed on the calibration of
the methods for such networks. The present document describes:
• methods for measurements of individual impairments or combinations of impairments to be made at acoustic
or electrical interfaces;
• methods for combining measures of different impairments into a single objective measure;
• methods for predicting the subjective effect of impairments that would be perceived by users.
The methods in the present document assume that subjects with normal hearing have been involved in the test.
Therefore, the instrumental methods estimate the perceived speech quality of persons with normal hearing. For each
method, the guide contains a general description to highlight the main points, and provides references for more detailed
information. The present document does not contain detailed specifications of the individual methods.
The present document concentrates on one-way speech quality in networks. It gives no guidance on how to evaluate
systems that include equipment such as echo cancellers or in which interactive impairments such as talker echo are
significant. The perceived quality in such cases depends not only on the one-way performance, but very much on the
behaviour of the equipment under duplex conditions; specifically, the influence of double-talk and delay needs to be
considered.
Although all assessments of overall speech quality are ultimately subjective because they depend on the user's opinion,
a distinction is made between:
• subjective methods, which involve real time user assessment; and
• objective methods, which use stored information on the user's assessment and therefore involve some degree
of calibration.
Objective methods for the evaluation of speech quality fall into three categories:
a) Comparison Methods: Methods based on the comparison of transmitted speech signal and a known reference.
b) Absolute Estimation Methods: Methods based on the absolute estimation of the speech quality (i.e. there is no
known reference signal); e.g. INMD (ITU-T Recommendation P.561 [i.16]).
c) Transmission Rating Models: Methods that derive a value for the expected speech quality from knowledge
about the network; e.g. ETSI Model (ETR 250 [i.1], ITU-T Recommendation G.107 [i.14]).
ETSI
---------------------- Page: 8 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
7 ETSI EG 201 377-1 V1.3.2 (2009-10)
The classification of assessment methods is depicted in figure 1.
Practical implementations of test equipment may include combinations of these methods. The focus of the present
document is on comparison methods (intrusive methods), which currently yield the most accurate results. The other
categories are only covered in short overviews, although they may be preferable for certain applications.
Subjective and Objective Methods
Users' subjective
assessment
Real - time Stored experience
assessment (i.e., past assessments)
Additional
experience
Subjective methods
Objective methods
Calibration
e.g., listening- only and conversational
Additional information
e.g.: radio link budget
error rate
cell loss rate
Signal - based measurements
Parameter- based models
e.g. speech samples and other test signals
Transmission Rating
Comparison Absolute Estimation
Models
Measure combined effect Predict combined effect of E - Model
of all impairments individually measured impairments
Talker Talker
Measure-
Parameters of:
ment Computa-
Network Network
"Talker",
Measure-
tion using
Network,
ment
knowledge
"Listener"
of network
Listener Listener
a) b) c)
Figure 1: Classification of assessment methods showing:
a) Comparison methods,
b) Absolute estimation methods,
c) Transmission rating models
NOTE: As an ETSI Guide, the present document provides guidelines for test methods that may be implemented.
However, a test method and especially quality models can only be applied in the way and within the
scope defined in the reference standard. A "Warning" indicates when this applies.
ETSI
---------------------- Page: 9 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
8 ETSI EG 201 377-1 V1.3.2 (2009-10)
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific.
• For a specific reference, subsequent revisions do not apply.
• Non-specific reference may be made only to a complete document or a part thereof and only in the following
cases:
- if it is accepted that it will be possible to use all future changes of the referenced document for the
purposes of the referring document;
- for informative references.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are indispensable for the application of the present document. For dated
references, only the edition cited applies. For non-specific references, the latest edition of the referenced document
(including any amendments) applies.
Not applicable.
2.2 Informative references
The following referenced documents are not essential to the use of the present document but they assist the user with
regard to a particular subject area. For non-specific references, the latest version of the referenced document (including
any amendments) applies.
[i.1] ETSI ETR 250: "Transmission and Multiplexing (TM); Speech communication quality from
mouth to ear for 3,1 kHz handset telephony across networks".
[i.2] ETSI EG 201 050: "Speech Processing, Transmission and Quality Aspects (STQ); Overall
Transmission Plan Aspects for Telephony in a Private Network".
[i.3] ETSI TR 102 082: "Speech Processing, Transmission and Quality Aspects (STQ); Guidance on
writing specifications and tests for non-linear and time variant telephony terminals".
[i.4] ETSI EG 202 396-1: "Speech and multimedia Transmission Quality (STQ); Speech quality
performance in the presence of background noise; Part 1: Background noise simulation technique
and background noise database".
[i.5] ETSI EG 202 396-2: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
quality performance in the presence of background noise; Part 2: Background noise transmission -
Network simulation - Subjective test database and results".
[i.6] ETSI EG 202 396-3: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
quality performance in the presence of background noise; Part 3: Background noise transmission -
Objective test methods".
[i.7] ETSI ES 202 737: "Speech and multimedia Transmission Quality (STQ); Transmission
requirements for narrowband VoIP terminals (handset and headset) from a QoS perspective as
perceived by the user".
ETSI
---------------------- Page: 10 ----------------------
SIST-V ETSI/ EG 201 377-1 V1.3.2:2010
9 ETSI EG 201 377-1 V1.3.2 (2009-10)
[i.8] ETSI ES 202 738: "Speech and multimedia Transmission Quality (STQ); Transmission
requirements for narrowband VoIP loudspeaking and handsfree terminals from a QoS perspective
as perceived by the user".
[i.9] ETSI ES 202 739: "Speech and multimediaTransmission Quality (STQ); Transmission
requirements for wideband VoIP terminals (handset and headset) from a QoS perspective as
perceived by the user".
[i.10] ETSI ES 202 740: "Speech and multimedia Transmission Quality (STQ); Transmission
requirements for wideband VoIP loudspeaking and handsfree terminals from a QoS perspective as
perceived by the user".
[i.11] EURESCOM Project P603 vol.1: "Quality of Service: Measurement Method Selection;
Deliverable 2: Measurement Method; Volume 1 of 2: Main Report".
[i.12] EURESCOM Project P603 vol.2: "Quality of Service: Measurement Method Selection;
Deliverable 2: Measurement Method; Volume 2 of 2: Annexes".
[i.13] ISO 532 (1975): "Acoustics - Method for calculating loudness level".
[i.14] ITU-T Recommendation G.107 (2002): "The E-model, a computational model for use in
transmission planning".
[i.15] ITU-T Recommendation P.501: "Test signals for use in telephonometry".
[i.16] ITU-T Recommendation P.561 (2002): "In-service, non-intrusive measurement device - voice
service measurements".
[i.17] ITU-T Recommendation P.562 (2004): "Analysis and interpretation of INMD voice-service
measurements".
[i.18] ITU-T Recommendation P.563 (2004): "Single-ended method for objective speech quality
assessment in narrow-band telephony applications".
[i.19] ITU-T Recommendation P.800 (1996): "Methods for subjective determination of transmission
quality".
[i.20] ITU-T Recommendation P.830 (1996): "Subjective performance assessment of telephone-band
and wideband digital codecs".
[i.21] ITU-T Recommendation P.835 (2003): " Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[i.22] ITU-T COM12-20: "Improvement of the P.861 perceptual speech quality measure".
[i.23] ITU-T COM12-24: "Proposed Annex A to Recommendation P.861".
[i.24] ITU-T COM12-34: "TOSQA - Telecommunication objective speech q
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.