SIST-V ETSI/EG 202 396-3 V1.2.1:2009
(Main)Speech Processing, Transmission and Quality Aspects (STQ) - Speech Quality performance in the presence of background noise - Part 3: Background noise transmission - Objective test methods
Speech Processing, Transmission and Quality Aspects (STQ) - Speech Quality performance in the presence of background noise - Part 3: Background noise transmission - Objective test methods
The present document aims to identify and define testing methodologies which can be used to objectively evaluate the performance of narrowband and wideband terminals and systems for speech communication in the presence of background noise. Background noise is a problem in mostly all situations and conditions and need to be taken into account in both, terminals and networks. The present document provides information about the testing methods applicable to objectively evaluate the speech quality in the presence of background noise. The present document includes:
-The description of the experts post evaluation process chosen to select the subjective test data being within the scope of the objective methods.
-The results of the performance evaluation of the currently existing methods described in ITU-T
Recommendation P.862 [i.16], [i.17] and in TOSQA2001 [i.19] which is chosen for the evaluation of terminals in the framework of ETSI VoIP speech quality test events [i.8], [i.9], [i.10] and [i.11].
-The method which is applicable to objectively determine the different parameters influencing the speech quality in the presence of background noise taking into account:
- the speech quality;
- the background noise transmission quality;
- the overall quality.
-The document is to be used in conjunction with:
- EG 202 396-1 [i.1] which describes a recording and reproduction setup for realistic simulation of background noise scenarios in lab-type environments for the performance evaluation of terminals and communication systems.
- EG 202 396-2 [i.2] which describes the simulation of network impairments and how to simulate realistic transmission network scenarios and which contains the methodology and results of the subjective scoring for the data forming the basis of the present document.
- French speech sentences as defined in ITU-T Recommendation P.501 [i.13] for wideband and English speech sentences as defined in ITU-T Recommendation P.501 [i.13] for narrowband.
Vidiki obdelave, prenosa in kakovosti govora (STQ) - Kakovost govora ob prisotnosti šuma ozadja - 3. del: Prenos šuma ozadja - Objektivne preskusne metode
General Information
Buy Standard
Standards Content (Sample)
ETSI EG 202 396-3 V1.2.1 (2009-01)
ETSI Guide
Speech Processing, Transmission and Quality Aspects (STQ);
Speech Quality performance
in the presence of background noise
Part 3: Background noise transmission -
Objective test methods
---------------------- Page: 1 ----------------------
2 ETSI EG 202 396-3 V1.2.1 (2009-01)
Reference
REG/STQ-00124
Keywords
noise, QoS, quality, speech
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2009.
All rights reserved.
TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
---------------------- Page: 2 ----------------------
3 ETSI EG 202 396-3 V1.2.1 (2009-01)
Contents
Intellectual Property Rights . 5
Foreword . 5
1 Scope . 6
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Abbreviations . 9
4 Speech signals to be used . 9
5 Selection of the data within the scope of the wideband objective model: Experts evaluation . 10
5.1 Selection process . 10
5.2 Results . 10
5.3 French database . 11
5.4 Czech database . 11
5.5 General differences between the databases . 13
6 Description of the wideband objective test method . 16
6.1 Introduction . 16
6.2 Speech sample preparation and nomenclature . 17
6.2.1 Speech sample preparation . 17
6.2.2 Nomenclature . 19
6.3 Principles of Relative Approach and Δ Relative Approach . 20
6.4 Objective N-MOS. 22
6.4.1 Introduction. 22
6.4.2 Description of N-MOS algorithm . 22
6.4.3 Comparing subjective and objective N-MOS results . 26
6.5 Objective S-MOS . 27
6.5.1 Introduction. 27
6.5.2 Description of S-MOS Algorithm . 28
6.5.3 Comparing Subjective and Objective S-MOS Results . 31
6.6 Objective G-MOS. 32
6.6.1 Description of G-MOS Algorithm . 32
6.6.2 Comparing subjective and objective G-MOS results . 33
6.7 Comparison of the objective method results for Czech and French samples . 33
6.8 Language Dependent Robustness of G-MOS . 38
7 Validation of the Wideband Objective Test Method . 39
7.1 Introduction . 39
7.2 All conditions results analysis . 41
7.2.1 Comparing subjective and objective N-MOS results . 41
7.2.2 Comparing subjective and objective S-MOS results . 42
7.2.3 Comparing Subjective and Objective G-MOS Results . 43
7.3 French Conditions Results Analysed . 43
7.3.1 Comparing Subjective and Objective N-MOS Results . 43
7.3.2 Comparing Subjective and Objective S-MOS Results . 44
7.3.3 Comparing subjective and objective G-MOS results . 45
7.4 Czech conditions results analysis . 45
7.4.1 Comparing subjective and objective N-MOS results . 45
7.4.2 Comparing subjective and objective S-MOS results . 46
7.4.3 Comparing Subjective and Objective G-MOS Results . 47
8 Objective Model for Narrowband Applications . 47
8.1 File pre-processing . 48
8.2 Adaptation of the Calculations . 48
ETSI
---------------------- Page: 3 ----------------------
4 ETSI EG 202 396-3 V1.2.1 (2009-01)
Annex A: Detailed post evaluation of listening test results . 50
Annex B: Results of PESQ and TOSQA2001 - Analysis of EG 202-396-2 database . 55
Annex C: Comparison of objective MOS versus auditory MOS for the All Data of Training
Period . 62
Annex D: Comparison of objective MOS versus auditory MOS for the Data not used during
the Training Period . 64
Annex E: Regression Coefficients for Czech data . 66
Annex F: Detailed STF 294 subjective and objective validation test results . 67
Annex G: Void . 71
Annex H: Extension of the EG 202 396-3 Speech Quality Test Method to Narrowband:
Adaptation, Training and Validation . 72
Annex I: Validation results of the modified EG 202 396-3 objective speech quality model for
narrowband data . 76
I.1 Introduction . 76
I.2 Description of the Databases . 76
I.3 Collection of the subjective scores . 77
I.4 Differences: HEAD acoustics training database vs. France Telecom validation databases . 79
I.5 Results . 80
I.6 Unmapped Results . 80
I.7 Mapped Results . 83
I.7.1 Use of mapping functions . 83
I.8 Conclusions . 89
History . 91
ETSI
---------------------- Page: 4 ----------------------
5 ETSI EG 202 396-3 V1.2.1 (2009-01)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality
(STQ).
The present document is a deliverable of ETSI Specialized Task Force (STF) 294 entitled: "Improving the quality of
eEurope wideband speech applications by developing a performance testing and evaluation methodology for
background noise transmission".
The present document is part 3 of a multi-part deliverable covering speech quality performance in the presence of
background noise, as identified below:
Part 1: "Background noise simulation technique and background noise database";
Part 2: "Background noise transmission - Network simulation - Subjective test database and results";
Part 3: "Background noise transmission - Objective test methods".
ETSI
---------------------- Page: 5 ----------------------
6 ETSI EG 202 396-3 V1.2.1 (2009-01)
1 Scope
The present document aims to identify and define testing methodologies which can be used to objectively evaluate the
performance of narrowband and wideband terminals and systems for speech communication in the presence of
background noise.
Background noise is a problem in mostly all situations and conditions and need to be taken into account in both,
terminals and networks. The present document provides information about the testing methods applicable to objectively
evaluate the speech quality in the presence of background noise. The present document includes:
• The description of the experts post evaluation process chosen to select the subjective test data being within the
scope of the objective methods.
• The results of the performance evaluation of the currently existing methods described in ITU-T
Recommendation P.862 [i.16], [i.17] and in TOSQA2001 [i.19] which is chosen for the evaluation of terminals
in the framework of ETSI VoIP speech quality test events [i.8], [i.9], [i.10] and [i.11].
• The method which is applicable to objectively determine the different parameters influencing the speech
quality in the presence of background noise taking into account:
- the speech quality;
- the background noise transmission quality;
- the overall quality.
• The document is to be used in conjunction with:
- EG 202 396-1 [i.1] which describes a recording and reproduction setup for realistic simulation of
background noise scenarios in lab-type environments for the performance evaluation of terminals and
communication systems.
- EG 202 396-2 [i.2] which describes the simulation of network impairments and how to simulate realistic
transmission network scenarios and which contains the methodology and results of the subjective scoring
for the data forming the basis of the present document.
- French speech sentences as defined in ITU-T Recommendation P.501 [i.13] for wideband and English
speech sentences as defined in ITU-T Recommendation P.501 [i.13] for narrowband.
ETSI
---------------------- Page: 6 ----------------------
7 ETSI EG 202 396-3 V1.2.1 (2009-01)
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific.
• For a specific reference, subsequent revisions do not apply.
• Non-specific reference may be made only to a complete document or a part thereof and only in the following
cases:
- if it is accepted that it will be possible to use all future changes of the referenced document for the
purposes of the referring document;
- for informative references.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are indispensable for the application of the present document. For dated
references, only the edition cited applies. For non-specific references, the latest edition of the referenced document
(including any amendments) applies.
Not applicable.
2.2 Informative references
The following referenced documents are not essential to the use of the present document but they assist the user with
regard to a particular subject area. For non-specific references, the latest version of the referenced document (including
any amendments) applies.
[i.1] ETSI EG 202 396-1: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
Quality performance in the presence of background noise; Part 1: Background Noise Simulation
Technique and Background Noise Database".
[i.2] ETSI EG 202 396-2: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
Quality performance in the presence of background noise; Part 2: Background Noise Transmission
- Network Simulation - Subjective Test Database and Results".
[i.3] ITU-T Recommendation P.835: "Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[i.4] ITU-T Recommendation P.800: "Methods for subjective determination of transmission quality".
[i.5] ITU-T Recommendation P.831: "Subjective performance evaluation of network echo cancellers".
[i.6] Genuit, K.: "Objective Evaluation of Acoustic Quality Based on a Relative Approach", InterNoise
'96, Liverpool, UK.
[i.7] ITU-T Recommendation SG 12 Contribution 34: "Evaluation of the quality of background noise
transmission using the "Relative Approach"".
[i.8] ETSI 2nd Speech Quality Test Event: "Anonymized Test Report", ETSI Plugtests, HEAD
acoustics, T-Systems Nova.
NOTE: Available at: http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.
Also available as ETSI TR 102 648-3.
ETSI
---------------------- Page: 7 ----------------------
8 ETSI EG 202 396-3 V1.2.1 (2009-01)
[i.9] ETSI 3rd Speech Quality Test Event: "Anonymized Test Report "IP Gateways"".
NOTE: Available at: http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.
[i.10] ETSI 3rd Speech Quality Test Event: "Anonymized Test Report "IP Phones"".
[i.11] ETSI 4th Speech Quality Test Event: "Anonymized Test Report "IP Gateways and IP Phones"".
NOTE: Available at: http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.
[i.12] F. Kettler, H.W. Gierlich, F. Rosenberger: "Application of the Relative Approach to Optimize
Packet Loss Concealment Implementations", DAGA, March 2003, Aachen, Germany.
[i.13] ITU-T Recommendation P.501: "Test Signals for Use in Telephonometry".
[i.14] R. Sottek, K. Genuit: "Models of Signal Processing in human hearing", International Journal of
Electronics and Communications (AEÜ)" vol. 59, 2005, p. 157-165.
NOTE: Available at: http://www.elsevier.de/aeue.
[i.15] SAE International - Document 2005-01-2513: "Tools and Methods for Product Sound Design of
Vehicles" R. Sottek, W. Krebber, G. Stanley.
[i.16] ITU-T Recommendation P.862: "Perceptual evaluation of speech quality (PESQ): An objective
method for end-to-end speech quality assessment of narrowband telephone networks and speech
codecs".
[i.17] ITU-T Recommendation P.862.1: "Mapping function for transforming P.862 raw result scores to
MOS-LQO".
[i.18] ITU-T Recommendation P.862.2: "Wideband extension to Recommendation P.862 for the
assessment of wideband telephone networks and speech codecs".
[i.19] ITU-T Recommendation SG 12 Contribution 19: "Results of objective speech quality assessment
of wideband speech using the Advanced TOSQA2001".
[i.20] ITU-T Recommendation G.722: "7 kHz audio-coding within 64 kbit/s".
[i.21] ITU-T Recommendation G.722.2: "Wideband coding of speech at around 16 kbit/s using Adaptive
Multi-Rate Wideband (AMR-WB)".
[i.22] ITU-T Recommendation P.56: "Objective measurement of active speech level".
[i.23] ITU-T Recommendation P.57: "Artificial ears".
[i.24] M. Spiegel: "Theory and problems of statistics", McGraw Hill, 1998.
[i.25] R.A. Fisher: "Statistical methods and scientific inference", Oliver and Boyd, 1956.
[i.26] M. Kendall: "Rank correlation methods", Charles Griffin & Company Limited, 1948.
[i.27] Sottek, R.: "Modelle zur Signalverarbeitung im menschlichen Gehör, PHD thesis RWTH Aachen,
1993".
[i.28] ITU-T Recommendation P.830: "Subjective performance assessment of telephone-band and
wideband digital codecs".
[i.29] ITU-T contribution COM 12-117, Study Period 1997-2000: "Report of the question 13/12
rapporteur's meeting (Solothurn, Germany, 6-10 March 2000)".
ETSI
---------------------- Page: 8 ----------------------
9 ETSI EG 202 396-3 V1.2.1 (2009-01)
3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ACR Absolute Comparison Rating
AMR Adaptive MultiRate
ASL Active Speech Level
NOTE: According to ITU-T Recommendation P.56 [i.22].
BGN BackGround Noise
CDF Cumulative Density Function
CI Confidence Interval
DB Data Base
dB SPL Sound Pressure Level re 20 µPa in dB
G-MOS Global MOS
NOTE: MOS related to the overall sample.
HP HighPass
IP Internet Protocol
IRS Intermediate Reference System
ITU International Telecommunication Union
ITU-T Telecom Standardization Body of ITU
MOS Mean Opinion Score
MOS-LQSN Mean Opinion Score - Listening Quality Subjective Noise
MRP Mouth Reference Point
NI Network I conditions
NII Network II conditions
NIII Network III conditions
NB NarrowBand
N-MOS Noise MOS
NOTE: MOS related to the noise transmission only.
NR Noise Reduction
NR (filter) Noise Reduction (filter)
PCM Pulse Code Modulation
PLC Packet Loss Concealment
RCV ReCeiVe
RMSE Random Mean Square Error
S-MOS Speech MOS
NOTE: MOS related to the speech signal only.
SNR Signal to Noise Ratio
STF Specialized Task Force
TOR Terms Of Reference
VAD Voice Activity Detection
VoIP Voice over IP
WB WideBand
4 Speech signals to be used
As with any objective model, the prediction of speech quality depends on the conditions under which the model was
tested and validated (see clauses 6.1 and 8). This dependency also applies to the speech material used in conjunction
with the objective model.
The wideband version of the model uses French speech sentences. The near end speech signal (clean speech signal)
consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can be
taken from ITU-T Recommendation P.501 [i.13].
ETSI
---------------------- Page: 9 ----------------------
10 ETSI EG 202 396-3 V1.2.1 (2009-01)
The narrowband version of the model uses English speech sentences. The near end speech signal (clean speech signal)
consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can be
taken from ITU-T Recommendation P.501 [i.13].
5 Selection of the data within the scope of the
wideband objective model: Experts evaluation
5.1 Selection process
The aim of the selection process was to identify those data in the databases described in EG 202 396-2 [i.2] which are
consistent with the scope of the objective models to be studied within the present document.
The experts were selected on the based on the definition found in ITU-T Recommendation e.g. P.831 [i.5]: experts are
experienced in subjective testing. Experts are able to describe an auditory event in detail and are able to separate
different events based on specific impairments. They are able to describe their subjective impressions in detail. They
have a background in technical implementations of noise reduction systems and transmission impairments and do have
detailed knowledge of the influence of particular implementations on subjective quality.
Their task was to select the relevant conditions within the scope of the model to be developed. Therefore they had to
verify the consistency of the data with respect to the following selection criteria:
1) Artefacts others than the ones which should have been produced by the signal processing described in [i.2]
e.g. due to the additional amplification required in order to provide a listening level of 79 dB SPL.
2) Inconsistencies within one condition due to the selection of the individual speech samples from the database
for subjective evaluation.
3) Inconsistencies within one condition due to statistical variation of the signal processing described in [i.2]
leading to non consistent judgements within this condition.
4) Inconsistencies due to ITU-T Recommendation P.56 [i.22] level adjustment process chosen for the complete
files including the background noise.
5) Impact of the different listening levels used in the two databases - the French and the Czech database.
As a result of the experts listening test a set of data was selected which is used for the development of the objective
model.
In the selection process five expert listeners (not native French/Czech speakers) were involved. Their task was not to
produce new judgements, but to check all the samples in the database with respect to the possible artefacts described
above.
A playback system with calibrated headphones was used for the test. The headphones used were Sennheiser HD 600
connected to the HEAD acoustics playback system HPS V. The equalization provided by the headphone manufacturer
was used since this was the one used in the French and Czech test setup.
All samples could be heard by the experts as often as required in order to get final agreement about the applicability of
the data within the terms of reference of the model. There was no limitation in comparing samples to the ones
previously heard.
5.2 Results
In general it could be observed that the 4 seconds sample size chosen in the experiment according to ITU-T
Recommendation P.835 [i.3] lead to a more difficult task even for expert listeners, especially in the case of non
stationary background noises. It is more difficult to identify the nature of the noise itself and then identify in addition
possible impairments introduced by the signal processing or by the network impairments. It is very likely that some
comparatively high standard deviations seen in the data are caused by these effects.
ETSI
---------------------- Page: 10 ----------------------
11 ETSI EG 202 396-3 V1.2.1 (2009-01)
5.3 French database
In general the French database is in line with the ToR except network condition NII. In network condition NII 1 %
packet loss was chosen which is too low for the conditions to be evaluated. Due to the inhomogeneously distributed
packet losses there are conditions where no packet loss is audible up to conditions where 5 out of 6 samples show
packet loss. Furthermore the packet loss may occur during speech as well as during the noise periods. The impact of the
different packet losses is not controlled with respect to their occurrence due to the statistical nature of the packet loss
distribution, even within a set of 6 samples used for evaluating one condition. Since packet loss is clearly audible under
NIII conditions (3 % packet loss) and much better distributed amongst the different samples the NII conditions are not
used within the scope of the objective method. They are either covered by the NI condition (0 % packet loss) or by the
NIII conditions. This results in 144 NII conditions which are not retained for the development of the model.
From the 288 NI and NIII conditions 28 conditions are not retained. The main reasons therefore are:
• Not consistent signal levels due to the amplification process.
• Insufficient S/N, speech almost inaudible.
The individual reasons for the samples of these conditions being not retained can be found in table A.1.
In total 260 out of 432 conditions are used as the reference for the objective model. In other wo
...
Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
ETSI Guide
Speech Processing, Transmission and Quality Aspects (STQ);
Speech Quality performance
in the presence of background noise
Part 3: Background noise transmission -
Objective test methods
---------------------- Page: 1 ----------------------
2 Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
Reference
REG/STQ-00124
Keywords
noise, QoS, quality, speech
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2008.
All rights reserved.
TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
ETSI
---------------------- Page: 2 ----------------------
3 Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
Contents
Intellectual Property Rights . 5
Foreword . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 7
2.2 Informative references . 7
3 Abbreviations . 8
4 Speech signals to be used . 9
5 Selection of the data within the scope of the wideband objective model: Experts evaluation . 9
5.1 Selection process . 9
5.2 Results . 10
5.3 French database . 10
5.4 Czech database . 11
5.5 General differences between the databases . 13
6 Description of the wideband objective test method . 16
6.1 Introduction . 16
6.2 Speech sample preparation and nomenclature . 17
6.2.1 Speech sample preparation . 17
6.2.2 Nomenclature . 19
6.3 Principles of Relative Approach and Δ Relative Approach . 20
6.4 Objective N-MOS. 22
6.4.1 Introduction. 22
6.4.2 Description of N-MOS algorithm . 22
6.4.3 Comparing subjective and objective N-MOS results . 26
6.5 Objective S-MOS . 27
6.5.1 Introduction. 27
6.5.2 Description of S-MOS Algorithm . 28
6.5.3 Comparing Subjective and Objective S-MOS Results . 31
6.6 Objective G-MOS. 32
6.6.1 Description of G-MOS Algorithm . 32
6.6.2 Comparing subjective and objective G-MOS results . 33
6.7 Comparison of the objective method results for Czech and French samples . 33
6.8 Language Dependent Robustness of G-MOS . 38
7 Validation of the Wideband Objective Test Method . 39
7.1 Introduction . 39
7.2 All conditions results analysis . 41
7.2.1 Comparing subjective and objective N-MOS results . 41
7.2.2 Comparing subjective and objective S-MOS results . 42
7.2.3 Comparing Subjective and Objective G-MOS Results . 43
7.3 French Conditions Results Analysed . 43
7.3.1 Comparing Subjective and Objective N-MOS Results . 43
7.3.2 Comparing Subjective and Objective S-MOS Results . 44
7.3.3 Comparing subjective and objective G-MOS results . 45
7.4 Czech conditions results analysis . 45
7.4.1 Comparing subjective and objective N-MOS results . 45
7.4.2 Comparing subjective and objective S-MOS results . 46
7.4.3 Comparing Subjective and Objective G-MOS Results . 47
8 Objective Model for Narrowband Applications . 47
8.1 File pre-processing . 48
8.2 Adaptation of the Calculations . 48
ETSI
---------------------- Page: 3 ----------------------
4 Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
Annex A: Detailed post evaluation of listening test results . 50
Annex B: Results of PESQ and TOSQA2001 - Analysis of EG 202-396-2 database . 55
Annex C: Comparison of objective MOS versus auditory MOS for the All Data of Training
Period . 62
Annex D: Comparison of objective MOS versus auditory MOS for the Data not used during
the Training Period . 64
Annex E: Regression Coefficients for Czech data . 66
Annex F: Detailed STF 294 subjective and objective validation test results . 67
Annex G: Void . 71
Annex H: Extension of the EG 202 396-3 Speech Quality Test Method to Narrowband:
Adaptation, Training and Validation . 72
Annex I: Validation results of the modified EG 202 396-3 objective speech quality model for
narrowband data . 77
I.1 Introduction . 77
I.2 Description of the Databases . 77
I.3 Collection of the subjective scores . 78
I.4 Differences: HEAD acoustics training database vs. France Telecom validation databases . 80
I.5 Results . 81
I.6 Unmapped Results . 81
I.7 Mapped Results . 84
I.7.1 Use of mapping functions . 84
Database #1 . 86
Database #2 . 87
Database #3 . 88
Database #4 . 89
I.8 Conclusions . 90
Annex J: Bibliography . 92
History . 93
ETSI
---------------------- Page: 4 ----------------------
5 Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech Processing, Transmission and Quality
Aspects (STQ), and is now submitted for the ETSI standards Membership Approval Procedure.
The present document is a deliverable of ETSI Specialized Task Force (STF) 294 entitled: "Improving the quality of
eEurope wideband speech applications by developing a performance testing and evaluation methodology for
background noise transmission".
The present document is part 3 of a multi-part deliverable covering speech quality performance in the presence of
background noise, as identified below:
Part 1: "Background noise simulation technique and background noise database";
Part 2: "Background noise transmission - Network simulation - Subjective test database and results";
Part 3: "Background noise transmission - Objective test methods".
ETSI
---------------------- Page: 5 ----------------------
6 Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
1 Scope
The present document aims to identify and define testing methodologies which can be used to objectively evaluate the
performance of narrowband and wideband terminals and systems for speech communication in the presence of
background noise.
Background noise is a problem in mostly all situations and conditions and need to be taken into account in both,
terminals and networks. The present document provides information about the testing methods applicable to objectively
evaluate the speech quality in the presence of background noise. The present document includes:
• The description of the experts post evaluation process chosen to select the subjective test data being within the
scope of the objective methods.
• The results of the performance evaluation of the currently existing methods described in ITU-T
Recommendation P.862 [i.16], [i.17] and in TOSQA2001 [i.19] which is chosen for the evaluation of terminals
in the framework of ETSI VoIP speech quality test events [i.8], [i.9], [i.10] and [i.11].
• The method which is applicable to objectively determine the different parameters influencing the speech
quality in the presence of background noise taking into account:
- the speech quality;
- the background noise transmission quality;
- the overall quality.
• The document is to be used in conjunction with:
- EG 202 396-1 [i.1] which describes a recording and reproduction setup for realistic simulation of
background noise scenarios in lab-type environments for the performance evaluation of terminals and
communication systems.
- EG 202 396-2 [i.2] which describes the simulation of network impairments and how to simulate realistic
transmission network scenarios and which contains the methodology and results of the subjective scoring
for the data forming the basis of the present document.
- French speech sentences as defined in ITU-T Recommendation P.501 [i.13] for wideband and English
speech sentences as defined in ITU-T Recommendation P.501 [i.13] for narrowband.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific.
• For a specific reference, subsequent revisions do not apply.
• Non-specific reference may be made only to a complete document or a part thereof and only in the following
cases:
- if it is accepted that it will be possible to use all future changes of the referenced document for the
purposes of the referring document;
- for informative references.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
ETSI
---------------------- Page: 6 ----------------------
7 Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
For online referenced documents, information sufficient to identify and locate the source shall be provided. Preferably,
the primary source of the referenced document should be cited, in order to ensure traceability. Furthermore, the
reference should, as far as possible, remain valid for the expected life of the document. The reference shall include the
method of access to the referenced document and the full network address, with the same punctuation and use of upper
case and lower case letters.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are indispensable for the application of the present document. For dated
references, only the edition cited applies. For non-specific references, the latest edition of the referenced document
(including any amendments) applies.
Not applicable.
2.2 Informative references
The following referenced documents are not essential to the use of the present document but they assist the user with
regard to a particular subject area. For non-specific references, the latest version of the referenced document (including
any amendments) applies.
[i.1] ETSI EG 202 396-1: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
Quality performance in the presence of background noise; Part 1: Background Noise Simulation
Technique and Background Noise Database".
[i.2] ETSI EG 202 396-2: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
Quality performance in the presence of background noise; Part 2: Background Noise Transmission
- Network Simulation - Subjective Test Database and Results".
[i.3] ITU-T Recommendation P.835: "Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[i.4] ITU-T Recommendation P.800: "Methods for subjective determination of transmission quality".
[i.5] ITU-T Recommendation P.831: "Subjective performance evaluation of network echo cancellers".
[i.6] Genuit, K.: "Objective Evaluation of Acoustic Quality Based on a Relative Approach", InterNoise
'96, Liverpool, UK.
[i.7] ITU-T Recommendation SG 12 Contribution 34: "Evaluation of the quality of background noise
transmission using the "Relative Approach"".
[i.8] ETSI 2nd Speech Quality Test Event: "Anonymized Test Report", ETSI Plugtests, HEAD
acoustics, T-Systems Nova.
NOTE: Available at: http://www.etsi.org/plugtests/History/2002VOIP2.htm. Also available as
ETSI TR 102 648-3.
[i.9] ETSI 3rd Speech Quality Test Event: "Anonymized Test Report "IP Gateways"".
NOTE: Available at: http://www.etsi.org/plugtests/History/2004SQTE.htm.
[i.10] ETSI 3rd Speech Quality Test Event: "Anonymized Test Report "IP Phones"".
[i.11] ETSI 4th Speech Quality Test Event: "Anonymized Test Report "IP Gateways and IP Phones"".
NOTE: Available at: http://www.etsi.org/plugtests/History/2006SQTE.htm.
[i.12] F. Kettler, H.W. Gierlich, F. Rosenberger: "Application of the Relative Approach to Optimize
Packet Loss Concealment Implementations", DAGA, March 2003, Aachen, Germany.
ETSI
---------------------- Page: 7 ----------------------
8 Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
[i.13] ITU-T Recommendation P.501: "Test Signals for Use in Telephonometry".
[i.14] R. Sottek, K. Genuit: "Models of Signal Processing in human hearing", International Journal of
Electronics and Communications (AEÜ)" vol. 59, 2005, p. 157-165.
NOTE: Available at: http://www.elsevier.de/aeue.
[i.15] SAE International - Document 2005-01-2513: "Tools and Methods for Product Sound Design of
Vehicles" R. Sottek, W. Krebber, G. Stanley.
[i.16] ITU-T Recommendation P.862: "Perceptual evaluation of speech quality (PESQ): An objective
method for end-to-end speech quality assessment of narrowband telephone networks and speech
codecs".
[i.17] ITU-T Recommendation P.862.1: "Mapping function for transforming P.862 raw result scores to
MOS-LQO".
[i.18] ITU-T Recommendation P.862.2: "Wideband extension to Recommendation P.862 for the
assessment of wideband telephone networks and speech codecs".
[i.19] ITU-T Recommendation SG 12 Contribution 19: "Results of objective speech quality assessment
of wideband speech using the Advanced TOSQA2001".
[i.20] ITU-T Recommendation G.722: "7 kHz audio-coding within 64 kbit/s".
[i.21] ITU-T Recommendation G.722.2: "Wideband coding of speech at around 16 kbit/s using Adaptive
Multi-Rate Wideband (AMR-WB)".
[i.22] ITU-T Recommendation P.56: "Objective measurement of active speech level".
[i.23] ITU-T Recommendation P.57: "Artificial ears".
[i.24] M. Spiegel: "Theory and problems of statistics", McGraw Hill, 1998.
[i.25] R.A. Fisher: "Statistical methods and scientific inference", Oliver and Boyd, 1956.
[i.26] M. Kendall: "Rank correlation methods", Charles Griffin & Company Limited, 1948.
[i.27] Sottek, R.: "Modelle zur Signalverarbeitung im menschlichen Gehör, PHD thesis RWTH Aachen,
1993".
[i.28] ITU-T Recommendation P.830: "Subjective performance assessment of telephone-band and
wideband digital codecs".
3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ACR Absolute Comparison Rating
AMR Adaptive MultiRate
ASL Active Speech Level
NOTE: According to ITU-T Recommendation P.56 [i.22].
BGN BackGround Noise
CDF Cumulative Density Function
CI Confidence Interval
DB Data Base
dB SPL Sound Pressure Level re 20 µPa in dB
G-MOS Global MOS
NOTE: MOS related to the overall sample.
ETSI
---------------------- Page: 8 ----------------------
9 Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
HP HighPass
IP Internet Protocol
IRS Intermediate Reference System
ITU International Telecommunication Union
ITU-T Telecom Standardization Body of ITU
MOS Mean Opinion Score
MOS-LQSN Mean Opinion Score - Listening Quality Subjective Noise
MRP Mouth Reference Point
NI Network I conditions
NII Network II conditions
NIII Network III conditions
NB NarrowBand
N-MOS Noise MOS
NOTE: MOS related to the noise transmission only.
NR Noise Reduction
NR (filter) Noise Reduction (filter)
PCM Pulse Code Modulation
PLC Packet Loss Concealment
RCV ReCeiVe
RMSE Random Mean Square Error
S-MOS Speech MOS
NOTE: MOS related to the speech signal only.
SNR Signal to Noise Ratio
STF Specialized Task Force
TOR Terms Of Reference
VAD Voice Activity Detection
VoIP Voice over IP
WB WideBand
4 Speech signals to be used
As with any objective model, the prediction of speech quality depends on the conditions under which the model was
tested and validated (see clauses 6.1 and 8). This dependency also applies to the speech material used in conjunction
with the objective model.
The wideband version of the model uses French speech sentences. The near end speech signal (clean speech signal)
consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can be
taken from ITU-T Recommendation P.501 [i.13].
The narrowband version of the model uses English speech sentences. The near end speech signal (clean speech signal)
consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can be
taken from ITU-T Recommendation P.501 [i.13].
5 Selection of the data within the scope of the
wideband objective model: Experts evaluation
5.1 Selection process
The aim of the selection process was to identify those data in the databases described in EG 202 396-2 [i.2] which are
consistent with the scope of the objective models to be studied within the present document.
ETSI
---------------------- Page: 9 ----------------------
10 Final draft ETSI EG 202 396-3 V1.2.1 (2008-11)
The experts were selected on the based on the definition found in ITU-T Recommendation e.g. P.831 [i.5]: experts are
experienced in subjective testing. Experts are able to describe an auditory event in detail and are able to separate
different events based on specific impairments. They are able to describe their subjective impressions in detail. They
have a background in technical implementations of noise reduction systems and transmission impairments and do have
detailed knowledge of the influence of particular implementations on subjective quality.
Their task was to select the relevant conditions within the scope of the model to be developed. Therefore they had to
verify the consistency of the data with respect to the following selection criteria:
1) Artefacts others than the ones which should have been produced by the signal processing described in [i.2]
e.g. due to the additional amplification required in order to provide a listening level of 79 dB SPL.
2) Inconsistencies within one condition due to the selection of the individual speech samples from the database
for subjective evaluation.
3) Inconsistencies within one condition due to statistical variation of the signal processing described in [i.2]
leading to non consistent judgements within this condition.
4) Inconsistencies due to ITU-T Recommendation P.56 [i.22] level adjustment process chosen for the complete
files including the background noise.
5) Impact of the different listening levels used in the two databases - the French and the Czech database.
As a result of the experts listening test a set of data was selected which is used for the development of the objective
model.
In the selection process five expert listeners (not native French/Czech speakers) were involved. Their task was not to
produce new judgements, but to check all the samples in the database with respect to the possible artefacts described
above.
A playback system with calibrated headphones was used for the test. The headphones used were Sennheiser HD 600
connected to the HEAD acoustics playback system HPS V. The equalization provided by the headphone manufacturer
was used since this was the one used in the French and Czech test setup.
All samples could be heard by the experts as often as required in order to get final agreement about the applicability of
the data within the terms of reference of the model. There was no limitation in comparing samples to the ones
previously heard.
5.2 Results
In general it could be observed that the 4 seconds sample size chosen in the experiment according to ITU-T
Recommendation P.835 [i.3] lead to a more difficult task even for expert listeners, especially in the case of non
stationary background noises. It is more difficult to identify the nature of the noise itself and then identify in addition
possible impairments introduced by the signal processing or by the network impairments. It is very likely that some
comparatively high standard deviations seen in the data are caused by these effects.
5.3 French database
In general the French database is in line with the ToR except network condition NII. In network condition NII 1 %
packet loss was chosen which is too low for the conditions to be evaluated. Due to the inhomogeneously distributed
packet losses there are conditions where no packet loss is audible up to conditions where 5 out of 6 samples show
packet loss. Furthermore the packet los
...
SLOVENSKI STANDARD
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
01-april-2009
Vidiki obdelave, prenosa in kakovosti govora (STQ) - Kakovost govora ob
prisotnosti šuma ozadja - 3. del: Prenos šuma ozadja - Objektivne preskusne
metode
Speech Processing, Transmission and Quality Aspects (STQ) - Speech Quality
performance in the presence of background noise - Part 3: Background noise
transmission - Objective test methods
Ta slovenski standard je istoveten z: EG 202 396-3 Version 1.2.1
ICS:
33.040.35 Telefonska omrežja Telephone networks
SIST-V ETSI/EG 202 396-3 V1.2.1:2009 en
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
---------------------- Page: 1 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
---------------------- Page: 2 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
ETSI EG 202 396-3 V1.2.1 (2009-01)
ETSI Guide
Speech Processing, Transmission and Quality Aspects (STQ);
Speech Quality performance
in the presence of background noise
Part 3: Background noise transmission -
Objective test methods
---------------------- Page: 3 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
2 ETSI EG 202 396-3 V1.2.1 (2009-01)
Reference
REG/STQ-00124
Keywords
noise, QoS, quality, speech
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2009.
All rights reserved.
TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
---------------------- Page: 4 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
3 ETSI EG 202 396-3 V1.2.1 (2009-01)
Contents
Intellectual Property Rights . 5
Foreword . 5
1 Scope . 6
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Abbreviations . 9
4 Speech signals to be used . 9
5 Selection of the data within the scope of the wideband objective model: Experts evaluation . 10
5.1 Selection process . 10
5.2 Results . 10
5.3 French database . 11
5.4 Czech database . 11
5.5 General differences between the databases . 13
6 Description of the wideband objective test method . 16
6.1 Introduction . 16
6.2 Speech sample preparation and nomenclature . 17
6.2.1 Speech sample preparation . 17
6.2.2 Nomenclature . 19
6.3 Principles of Relative Approach and Δ Relative Approach . 20
6.4 Objective N-MOS. 22
6.4.1 Introduction. 22
6.4.2 Description of N-MOS algorithm . 22
6.4.3 Comparing subjective and objective N-MOS results . 26
6.5 Objective S-MOS . 27
6.5.1 Introduction. 27
6.5.2 Description of S-MOS Algorithm . 28
6.5.3 Comparing Subjective and Objective S-MOS Results . 31
6.6 Objective G-MOS. 32
6.6.1 Description of G-MOS Algorithm . 32
6.6.2 Comparing subjective and objective G-MOS results . 33
6.7 Comparison of the objective method results for Czech and French samples . 33
6.8 Language Dependent Robustness of G-MOS . 38
7 Validation of the Wideband Objective Test Method . 39
7.1 Introduction . 39
7.2 All conditions results analysis . 41
7.2.1 Comparing subjective and objective N-MOS results . 41
7.2.2 Comparing subjective and objective S-MOS results . 42
7.2.3 Comparing Subjective and Objective G-MOS Results . 43
7.3 French Conditions Results Analysed . 43
7.3.1 Comparing Subjective and Objective N-MOS Results . 43
7.3.2 Comparing Subjective and Objective S-MOS Results . 44
7.3.3 Comparing subjective and objective G-MOS results . 45
7.4 Czech conditions results analysis . 45
7.4.1 Comparing subjective and objective N-MOS results . 45
7.4.2 Comparing subjective and objective S-MOS results . 46
7.4.3 Comparing Subjective and Objective G-MOS Results . 47
8 Objective Model for Narrowband Applications . 47
8.1 File pre-processing . 48
8.2 Adaptation of the Calculations . 48
ETSI
---------------------- Page: 5 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
4 ETSI EG 202 396-3 V1.2.1 (2009-01)
Annex A: Detailed post evaluation of listening test results . 50
Annex B: Results of PESQ and TOSQA2001 - Analysis of EG 202-396-2 database . 55
Annex C: Comparison of objective MOS versus auditory MOS for the All Data of Training
Period . 62
Annex D: Comparison of objective MOS versus auditory MOS for the Data not used during
the Training Period . 64
Annex E: Regression Coefficients for Czech data . 66
Annex F: Detailed STF 294 subjective and objective validation test results . 67
Annex G: Void . 71
Annex H: Extension of the EG 202 396-3 Speech Quality Test Method to Narrowband:
Adaptation, Training and Validation . 72
Annex I: Validation results of the modified EG 202 396-3 objective speech quality model for
narrowband data . 76
I.1 Introduction . 76
I.2 Description of the Databases . 76
I.3 Collection of the subjective scores . 77
I.4 Differences: HEAD acoustics training database vs. France Telecom validation databases . 79
I.5 Results . 80
I.6 Unmapped Results . 80
I.7 Mapped Results . 83
I.7.1 Use of mapping functions . 83
I.8 Conclusions . 89
History . 91
ETSI
---------------------- Page: 6 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
5 ETSI EG 202 396-3 V1.2.1 (2009-01)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality
(STQ).
The present document is a deliverable of ETSI Specialized Task Force (STF) 294 entitled: "Improving the quality of
eEurope wideband speech applications by developing a performance testing and evaluation methodology for
background noise transmission".
The present document is part 3 of a multi-part deliverable covering speech quality performance in the presence of
background noise, as identified below:
Part 1: "Background noise simulation technique and background noise database";
Part 2: "Background noise transmission - Network simulation - Subjective test database and results";
Part 3: "Background noise transmission - Objective test methods".
ETSI
---------------------- Page: 7 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
6 ETSI EG 202 396-3 V1.2.1 (2009-01)
1 Scope
The present document aims to identify and define testing methodologies which can be used to objectively evaluate the
performance of narrowband and wideband terminals and systems for speech communication in the presence of
background noise.
Background noise is a problem in mostly all situations and conditions and need to be taken into account in both,
terminals and networks. The present document provides information about the testing methods applicable to objectively
evaluate the speech quality in the presence of background noise. The present document includes:
• The description of the experts post evaluation process chosen to select the subjective test data being within the
scope of the objective methods.
• The results of the performance evaluation of the currently existing methods described in ITU-T
Recommendation P.862 [i.16], [i.17] and in TOSQA2001 [i.19] which is chosen for the evaluation of terminals
in the framework of ETSI VoIP speech quality test events [i.8], [i.9], [i.10] and [i.11].
• The method which is applicable to objectively determine the different parameters influencing the speech
quality in the presence of background noise taking into account:
- the speech quality;
- the background noise transmission quality;
- the overall quality.
• The document is to be used in conjunction with:
- EG 202 396-1 [i.1] which describes a recording and reproduction setup for realistic simulation of
background noise scenarios in lab-type environments for the performance evaluation of terminals and
communication systems.
- EG 202 396-2 [i.2] which describes the simulation of network impairments and how to simulate realistic
transmission network scenarios and which contains the methodology and results of the subjective scoring
for the data forming the basis of the present document.
- French speech sentences as defined in ITU-T Recommendation P.501 [i.13] for wideband and English
speech sentences as defined in ITU-T Recommendation P.501 [i.13] for narrowband.
ETSI
---------------------- Page: 8 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
7 ETSI EG 202 396-3 V1.2.1 (2009-01)
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific.
• For a specific reference, subsequent revisions do not apply.
• Non-specific reference may be made only to a complete document or a part thereof and only in the following
cases:
- if it is accepted that it will be possible to use all future changes of the referenced document for the
purposes of the referring document;
- for informative references.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are indispensable for the application of the present document. For dated
references, only the edition cited applies. For non-specific references, the latest edition of the referenced document
(including any amendments) applies.
Not applicable.
2.2 Informative references
The following referenced documents are not essential to the use of the present document but they assist the user with
regard to a particular subject area. For non-specific references, the latest version of the referenced document (including
any amendments) applies.
[i.1] ETSI EG 202 396-1: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
Quality performance in the presence of background noise; Part 1: Background Noise Simulation
Technique and Background Noise Database".
[i.2] ETSI EG 202 396-2: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
Quality performance in the presence of background noise; Part 2: Background Noise Transmission
- Network Simulation - Subjective Test Database and Results".
[i.3] ITU-T Recommendation P.835: "Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[i.4] ITU-T Recommendation P.800: "Methods for subjective determination of transmission quality".
[i.5] ITU-T Recommendation P.831: "Subjective performance evaluation of network echo cancellers".
[i.6] Genuit, K.: "Objective Evaluation of Acoustic Quality Based on a Relative Approach", InterNoise
'96, Liverpool, UK.
[i.7] ITU-T Recommendation SG 12 Contribution 34: "Evaluation of the quality of background noise
transmission using the "Relative Approach"".
[i.8] ETSI 2nd Speech Quality Test Event: "Anonymized Test Report", ETSI Plugtests, HEAD
acoustics, T-Systems Nova.
NOTE: Available at: http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.
Also available as ETSI TR 102 648-3.
ETSI
---------------------- Page: 9 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
8 ETSI EG 202 396-3 V1.2.1 (2009-01)
[i.9] ETSI 3rd Speech Quality Test Event: "Anonymized Test Report "IP Gateways"".
NOTE: Available at: http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.
[i.10] ETSI 3rd Speech Quality Test Event: "Anonymized Test Report "IP Phones"".
[i.11] ETSI 4th Speech Quality Test Event: "Anonymized Test Report "IP Gateways and IP Phones"".
NOTE: Available at: http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.
[i.12] F. Kettler, H.W. Gierlich, F. Rosenberger: "Application of the Relative Approach to Optimize
Packet Loss Concealment Implementations", DAGA, March 2003, Aachen, Germany.
[i.13] ITU-T Recommendation P.501: "Test Signals for Use in Telephonometry".
[i.14] R. Sottek, K. Genuit: "Models of Signal Processing in human hearing", International Journal of
Electronics and Communications (AEÜ)" vol. 59, 2005, p. 157-165.
NOTE: Available at: http://www.elsevier.de/aeue.
[i.15] SAE International - Document 2005-01-2513: "Tools and Methods for Product Sound Design of
Vehicles" R. Sottek, W. Krebber, G. Stanley.
[i.16] ITU-T Recommendation P.862: "Perceptual evaluation of speech quality (PESQ): An objective
method for end-to-end speech quality assessment of narrowband telephone networks and speech
codecs".
[i.17] ITU-T Recommendation P.862.1: "Mapping function for transforming P.862 raw result scores to
MOS-LQO".
[i.18] ITU-T Recommendation P.862.2: "Wideband extension to Recommendation P.862 for the
assessment of wideband telephone networks and speech codecs".
[i.19] ITU-T Recommendation SG 12 Contribution 19: "Results of objective speech quality assessment
of wideband speech using the Advanced TOSQA2001".
[i.20] ITU-T Recommendation G.722: "7 kHz audio-coding within 64 kbit/s".
[i.21] ITU-T Recommendation G.722.2: "Wideband coding of speech at around 16 kbit/s using Adaptive
Multi-Rate Wideband (AMR-WB)".
[i.22] ITU-T Recommendation P.56: "Objective measurement of active speech level".
[i.23] ITU-T Recommendation P.57: "Artificial ears".
[i.24] M. Spiegel: "Theory and problems of statistics", McGraw Hill, 1998.
[i.25] R.A. Fisher: "Statistical methods and scientific inference", Oliver and Boyd, 1956.
[i.26] M. Kendall: "Rank correlation methods", Charles Griffin & Company Limited, 1948.
[i.27] Sottek, R.: "Modelle zur Signalverarbeitung im menschlichen Gehör, PHD thesis RWTH Aachen,
1993".
[i.28] ITU-T Recommendation P.830: "Subjective performance assessment of telephone-band and
wideband digital codecs".
[i.29] ITU-T contribution COM 12-117, Study Period 1997-2000: "Report of the question 13/12
rapporteur's meeting (Solothurn, Germany, 6-10 March 2000)".
ETSI
---------------------- Page: 10 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
9 ETSI EG 202 396-3 V1.2.1 (2009-01)
3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ACR Absolute Comparison Rating
AMR Adaptive MultiRate
ASL Active Speech Level
NOTE: According to ITU-T Recommendation P.56 [i.22].
BGN BackGround Noise
CDF Cumulative Density Function
CI Confidence Interval
DB Data Base
dB SPL Sound Pressure Level re 20 µPa in dB
G-MOS Global MOS
NOTE: MOS related to the overall sample.
HP HighPass
IP Internet Protocol
IRS Intermediate Reference System
ITU International Telecommunication Union
ITU-T Telecom Standardization Body of ITU
MOS Mean Opinion Score
MOS-LQSN Mean Opinion Score - Listening Quality Subjective Noise
MRP Mouth Reference Point
NI Network I conditions
NII Network II conditions
NIII Network III conditions
NB NarrowBand
N-MOS Noise MOS
NOTE: MOS related to the noise transmission only.
NR Noise Reduction
NR (filter) Noise Reduction (filter)
PCM Pulse Code Modulation
PLC Packet Loss Concealment
RCV ReCeiVe
RMSE Random Mean Square Error
S-MOS Speech MOS
NOTE: MOS related to the speech signal only.
SNR Signal to Noise Ratio
STF Specialized Task Force
TOR Terms Of Reference
VAD Voice Activity Detection
VoIP Voice over IP
WB WideBand
4 Speech signals to be used
As with any objective model, the prediction of speech quality depends on the conditions under which the model was
tested and validated (see clauses 6.1 and 8). This dependency also applies to the speech material used in conjunction
with the objective model.
The wideband version of the model uses French speech sentences. The near end speech signal (clean speech signal)
consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can be
taken from ITU-T Recommendation P.501 [i.13].
ETSI
---------------------- Page: 11 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
10 ETSI EG 202 396-3 V1.2.1 (2009-01)
The narrowband version of the model uses English speech sentences. The near end speech signal (clean speech signal)
consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can be
taken from ITU-T Recommendation P.501 [i.13].
5 Selection of the data within the scope of the
wideband objective model: Experts evaluation
5.1 Selection process
The aim of the selection process was to identify those data in the databases described in EG 202 396-2 [i.2] which are
consistent with the scope of the objective models to be studied within the present document.
The experts were selected on the based on the definition found in ITU-T Recommendation e.g. P.831 [i.5]: experts are
experienced in subjective testing. Experts are able to describe an auditory event in detail and are able to separate
different events based on specific impairments. They are able to describe their subjective impressions in detail. They
have a background in technical implementations of noise reduction systems and transmission impairments and do have
detailed knowledge of the influence of particular implementations on subjective quality.
Their task was to select the relevant conditions within the scope of the model to be developed. Therefore they had to
verify the consistency of the data with respect to the following selection criteria:
1) Artefacts others than the ones which should have been produced by the signal processing described in [i.2]
e.g. due to the additional amplification required in order to provide a listening level of 79 dB SPL.
2) Inconsistencies within one condition due to the selection of the individual speech samples from the database
for subjective evaluation.
3) Inconsistencies within one condition due to statistical variation of the signal processing described in [i.2]
leading to non consistent judgements within this condition.
4) Inconsistencies due to ITU-T Recommendation P.56 [i.22] level adjustment process chosen for the complete
files including the background noise.
5) Impact of the different listening levels used in the two databases - the French and the Czech database.
As a result of the experts listening test a set of data was selected which is used for the development of the objective
model.
In the selection process five expert listeners (not native French/Czech speakers) were involved. Their task was not to
produce new judgements, but to check all the samples in the database with respect to the possible artefacts described
above.
A playback system with calibrated headphones was used for the test. The headphones used were Sennheiser HD 600
connected to the HEAD acoustics playback system HPS V. The equalization provided by the headphone manufacturer
was used since this was the one used in the French and Czech test setup.
All samples could be heard by the experts as often as required in order to get final agreement about the applicability of
the data within the terms of reference of the model. There was no limitation in comparing samples to the ones
previously heard.
5.2 Results
In general it could be observed that the 4 seconds sample size chosen in the experiment according to ITU-T
Recommendation P.835 [i.3] lead to a more difficult task even for expert listeners, especially in the case of non
stationary background noises. It is more difficult to identify the nature of the noise itself and then identify in addition
possible impairments introduced by the signal processing or by the network impairments. It is very likely that some
comparatively high standard deviations seen in the data are caused by these effects.
ETSI
---------------------- Page: 12 ----------------------
SIST-V ETSI/EG 202 396-3 V1.2.1:2009
11 ETSI EG 202 396-3 V1.2.1 (2009-01)
5.3 French database
In general the French database is in line with the ToR except network condition NII. In network condition NII 1 %
packet loss was chosen which is too low for the conditions to be evaluated.
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.