ETSI TS 103 106 V1.5.1 (2018-04)

Speech and multimedia Transmission Quality (STQ); Speech quality performance in the presence of background noise: Background noise transmission for mobile terminals-objective test methods

RTS/STQ-271

General Information

• Status: Published
• Publication Date: 10-Apr-2018
• Technical Committee: STQ - Speech and multimedia Transmission Quality
• Current Stage: 12 - Completion
• Due Date: 23-Apr-2018
• Completion Date: 11-Apr-2018

Standards Content (Sample)

ETSI TS 103 106 V1.5.1 (2018-04)






TECHNICAL SPECIFICATION
Speech and multimedia Transmission Quality (STQ);
Speech quality performance in
the presence of background noise:
Background noise transmission for
mobile terminals-objective test methods

---------------------- Page: 1 ----------------------
2 ETSI TS 103 106 V1.5.1 (2018-04)



Reference
RTS/STQ-271
Keywords
noise, quality, speech, testing, transmission
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2018.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M logo is protected for the benefit of its Members.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI TS 103 106 V1.5.1 (2018-04)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 7
3 Abbreviations . 8
4 Introduction . 9
5 Underlying speech databases and preparations . 9
6 Modifications to the model described in ETSI EG 202 396-3 . 10
6.1 Prefiltering in Narrowband Mode (NB) . 10
6.2 Void . 10
6.3 Speech level adjustment in wideband . 11
6.4 Modified neural network for S-MOS . 11
6.5 Retraining of parameter regression for N-MOS and G-MOS . 12
7 Comparison of objective and subjective results after the training process . 13
7.0 General . 13
7.1 Results in wideband mode . 13
7.1.0 General . 13
7.1.1 Results for database "Audience - Test 3" . 14
7.1.2 Results for database "Audience - Test 3L" (excluded during retraining) . 14
7.1.3 Results for database "Audience - Test 4" . 15
7.1.4 Results for database "Audience - Test 4L" . 16
7.1.5 Results for database "Nokia - Test 1" . 16
7.1.6 Results for database "Nokia - Test 2" (excluded during retraining) . 17
7.1.7 Results for database "Orange" . 18
7.1.8 Results for database "Qualcomm - Test 3" . 18
7.1.9 Results for database "Qualcomm - Test 4" . 19
7.2 Results in narrowband mode . 19
7.2.0 General . 19
7.2.1 Results for database "Audience - Test 1" . 20
7.2.2 Results for database "Audience - Test 1L" . 20
7.2.3 Results for database "Audience - Test 2" . 21
7.2.4 Results for database "Audience - Test 2L" . 22
7.2.5 Results for database "Qualcomm- Test 1" . 22
7.2.6 Results for database "Qualcomm- Test 2" . 23
8 Validation results . 23
8.0 Preamble . 23
8.1 Audience validation data . 24
8.1.1 Description of tests . 24
8.1.2 Description of validation results . 25
8.1.2.0 General explanation . 25
8.1.2.1 Experiment 5: Narrowband . 25
8.1.2.2 Experiment 6: Narrowband . 27
8.1.2.3 Experiment 7: Wideband . 29
8.1.2.4 Experiment 8: Wideband . 31
8.2 Orange validation data . 33
8.2.1 Description of tests . 33
8.2.2 Description of validation results . 34
8.3 Qualcomm validation data . 35
8.3.1 Description of tests . 35
ETSI

---------------------- Page: 3 ----------------------
4 ETSI TS 103 106 V1.5.1 (2018-04)
8.3.2 Description of validation results . 38
8.4 Validation data for additional use cases . 42
8.4.1 Tests 1 & 2: Description . 42
8.4.2 Tests 1 & 2: Results . 43
8.4.3 Tests 3, 4, 5 & 6: Description . 52
8.4.4 Tests 3 & 4: Results, Narrowband . 53
8.4.5 Tests 5 & 6: Results, Wideband . 61
9 Application of the retrained model . 68
Annex A (normative): Summary of Retraining Databases. 69
Annex B (normative): Test vectors for model verification . 70
B.0 Test vectors . 70
B.1 Audience test vectors. 70
B.2 Orange test vectors . 72
Annex C (normative): Speech material to be used for objective testing . 73
Annex D (informative): Subjective testing framework used for the present document . 74
D.1 Introduction . 74
D.2 Subjective test plan . 74
D.2.1 Traceability. 74
D.2.2 Speech database requirements . 74
D.2.3 Reference Conditions . 74
D.2.4 Test Conditions . 74
D.2.5 Pre-processing of reference conditions. 76
D.2.6 Post-processing of test conditions . 76
D.2.7 Calibration and equalization of headphones for presentation . 76
D.2.8 Requirements on the listening laboratory . 76
D.2.9 Experimental design . 76
D.2.10 Training session . 77
D.3 Set-up for acquisition of test conditions . 77
D.3.1 Terminal positioning and HATS calibration . 77
D.3.2 Background Noise reproduction . 77
D.3.3 Noise and speech playback synchronization . 78
D.3.4 Convergence sequence . 78
D.3.5 Example of noise and speech playback sequence including convergence period . 78
D.3.6 Recordings at the network simulator electrical reference point . 79
D.3.7 Recordings at the MRP and terminal's primary microphone location . 79
D.4 Processing test plan block diagram . 80
History . 82

ETSI

---------------------- Page: 4 ----------------------
5 ETSI TS 103 106 V1.5.1 (2018-04)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Specification (TS) has been produced by ETSI Technical Committee Speech and multimedia
Transmission Quality (STQ).
The present document is to be used in conjunction with the ETSI ES 202 396-1 [i.2] and ETSI EG 202 396-3 [i.4]:
ETSI ES 202 396-1: "Background noise simulation technique and background noise database";
ETSI EG 202 396-3: "Background noise transmission - Objective test methods".
The present document is based on the objective test method described in ETSI EG 202 396-3 [i.4] and contains
modifications of the model required in order to provide a good prediction of the uplink speech quality in the presence of
background noise of modern mobile terminals.
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI

---------------------- Page: 5 ----------------------
6 ETSI TS 103 106 V1.5.1 (2018-04)
1 Scope
The present document describes testing methodologies which can be used to objectively evaluate the performance of
narrowband and wideband mobile terminals for speech communication in the presence of background noise.
Background noise is a problem in mostly all situations and conditions and needs to be taken into account in both,
terminals and networks. The present document provides information about the testing methods applicable to objectively
evaluate the speech quality of mobile terminals with AMR and AMR-WB codecs in the presence of background noise.
The present document includes:
• The method which is applicable to objectively determine the different parameters influencing the speech
quality in the presence of background noise taking into account:
- the speech quality;
- the background noise transmission quality;
- the overall quality.
• The description of the adaptation of the test method described in ETSI ES 202 396-1 [i.2].
• The model results in comparison with the underlying subjective tests used for the retraining of the objective
model.
• The model validation results:
- Additional validation results are provided for cases which include some conditions outside the scope of
ETSI ES 202 396-1 [i.2]. These include music as background noise, and user holding a handset in other
than nominal position, as defined in Recommendation ITU-T P.64 [i.24]. In addition, validation results
are provided for Chinese language.
The present document is to be used in conjunction with:
- ETSI ES 202 396-1 [i.2] which describes a recording and reproduction setup for realistic simulation of
background noise scenarios in lab-type environments for the performance evaluation of terminals and
communication systems.
- ETSI EG 202 396-3 [i.4] which describes the basic objective model underlying to the Model described in the
present document.
- American English speech sentences as enclosed in the present document.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference/.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
Not applicable.
ETSI

---------------------- Page: 6 ----------------------
7 ETSI TS 103 106 V1.5.1 (2018-04)
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] 3GPP S4-120542: "Common subjective testing framework for training of P.835 test predictors".
[i.2] ETSI ES 202 396-1: "Speech and multimedia Transmission Quality (STQ); Speech quality
performance in the presence of background noise; Part 1: Background noise simulation technique
and background noise database".
[i.3] Void.
[i.4] ETSI EG 202 396-3: "Speech and multimedia Transmission Quality (STQ); Speech Quality
performance in the presence of background noise Part 3: Background noise transmission -
Objective test methods".
[i.5] ETSI TS 126 073: "Digital cellular telecommunications system (Phase 2+) (GSM); Universal
Mobile Telecommunications System (UMTS); LTE; ANSI-C code for the Adaptive Multi Rate
(AMR) speech codec (3GPP TS 26.073)".
[i.6] Recommendation ITU-T P.835: "Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[i.7] Recommendation ITU-T G.722.2: "Wideband coding of speech at around 16 kbit/s using Adaptive
Multi-Rate Wideband (AMR-WB)".
[i.8] Recommendation ITU-T P.56: "Objective measurement of active speech level".
[i.9] Recommendation ITU-T P.1401: "Methods, metrics and procedures for statistical evaluation,
qualifying and comparison of objective quality prediction models".
[i.10] Void.
[i.11] Recommendation ITU-T G.191: "Software tools for speech and audio coding standardization".
[i.12] Void.
[i.13] Recommendation ITU-T P.501: "Test Signals for Use in Telephonometry".
[i.14] Recommendation ITU-T P.58: "Head and Torso simulator for telephonometry".
[i.15] Recommendation ITU-T P.57: "Artificial ears".
[i.16] ETSI TS 126 131: "Universal Mobile Telecommunications System (UMTS); LTE; Terminal
acoustic characteristics for telephony; Requirements (3GPP TS 26.131)".
[i.17] Recommendation ITU-T P.800: "Methods for subjective determination of transmission quality".
[i.18] ETSI TS 126 132: "Universal Mobile Telecommunications System (UMTS); LTE; Speech and
video telephony terminal acoustic test specification (3GPP TS 26.132)".
[i.19] Void.
[i.20] Recommendation ITU-T TD 477 (GEN/12): "Handbook of subjective test practical procedures"
(temporary document) - Geneva, 18-27 January 2011.
[i.21] AH-11-029: "Better Reference System for the P.835 SIG Rating Scale", Q7/12 Rapporteur's
meeting, 20-21 June 2011, Geneva, Switzerland.
ETSI

---------------------- Page: 7 ----------------------
8 ETSI TS 103 106 V1.5.1 (2018-04)
[i.22] 3GPP, Tdoc S4(12)0621, Ext-ATS Permanent document (EATS-3): "Common subjective testing
framework for validation of P.835 test predictors".
[i.23] Recommendation ITU-T P.50: "Artificial voices".
[i.24] Recommendation ITU-T P.64: "Determination of sensitivity/frequency characteristics of local
telephone systems".
3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
78KBP 7,8 kHz band-pass according to Recommendation ITU-T G.191
AMR Adaptive MultiRate
AMR-NB Adaptive Multirate Codec - Narrow Band
AMR-WB Adaptive Multi-Rate Wideband Speech Codec
BAK Background Noise Component
dB SPL Sound Pressure Level re 20 µPa in dB
DRP Drum Reference Point
DTX Discontinuous Transmission
EATS Enhanced Acoustic Test Specification
EXP Experiment
FB Fullband
G-MOS Global MOS
NOTE: MOS related to the overall sample.
HATS Head And Torso Simulator
HHHF Hand-Held Hands-Free
IRS Intermediate Reference System
ITU International Telecommunication Union
ITU-T Telecommunication Standardization Sector of ITU
MOS Mean Opinion Score
MRP Mouth Reference Point
MSIN Mobile Station Input Filter
NB Narrowband
N-MOS Noise MOS
NOTE: MOS related to the noise transmission only.
NS Noise Suppression
NTT Nippon Telegraph and Telephone
OVRL Overall (speech + noise) Component
PRO Professional
RCV ReCeiVe
RMSE Root Mean Square Error
RMSE* epsilon insensitive Root Mean Square Error
SIG SIGnal component
S-MOS Speech MOS
NOTE: MOS related to the speech signal only.
SND Sending Direction
SNR Signal to Noise Ratio
SPL Sound Pressure Level
WB Wideband
WCDMA Wideband Code Division Multiple Access
ETSI

---------------------- Page: 8 ----------------------
9 ETSI TS 103 106 V1.5.1 (2018-04)
4 Introduction
The present document describes the modifications of the ETSI EG 202 396-3 [i.4] model which were necessary to adapt
to the training databases provided by the 3GPP contributors listed in annex A. The core model itself retains mainly
unmodified except the points given in the clauses below. Modifications affect the narrow- and wideband mode in
different ways.
The adapted objective method described in the present document is intended to be used for all types of modern mobile
terminals using different bitrates of AMR [i.5] and AMR-WB [i.7] coding.
5 Underlying speech databases and preparations
The base for each mode of the objective model (wideband/narrowband) as described in ETSI EG 202 396-3 [i.4] are
listening tests conducted according to Recommendation ITU-T P.835 [i.6]. From the beginning of the development,
these listening test databases were designed to be a training set for predicting Recommendation ITU-T P.835 [i.6]
scores. They included a huge amount of conditions (> 170) and a wide range of speech and noise quality. Besides real
terminals also terminal simulations and transmission impairments were included. However, the data and processing
included were based on technologies actual at the time when the standard and its updates were created.
The underlying databases for the retraining as described in the present document were created using real state-of-the-art
mobile devices and thus the quality ranges yielded may not be normally distributed over all MOS scales. The context
between the databases can also differ (e.g. pure handset recordings vs. mixed handset/hands-free databases).
Furthermore new reference conditions extensively discussed in different standards groups and described in [i.1] were
included in the tests.
Table 1: Set of reference conditions
File SIG. SNR Noise Type
i01 Source (filtered) No Noise -
i02 Source (filtered) 0 dB Fullsize_Car1_130Kmh_binaural
i03 Source (filtered) 12 dB Fullsize_Car1_130Kmh_binaural
i04 Source (filtered) 24 dB Fullsize_Car1_130Kmh_binaural
i05 Source (filtered) 36 dB Fullsize_Car1_130Kmh_binaural
i06 NS Level 1 No Noise -
i07 NS Level 2 No Noise -
i08 NS Level 3 No Noise -
i09 NS Level 4 No Noise -
i10 NS Level 3 24 dB Fullsize_Car1_130Kmh_binaural
i11 NS Level 2 12 dB Fullsize_Car1_130Kmh_binaural
i12 NS Level 1 0 dB Fullsize_Car1_130Kmh_binaural
NOTE: In case of clipping is generated for condition i02 and/or i12, a different signal
scaling is recommended (e.g. 73 dB refers to -36 instead of -26 dBov).
SPL

Each training database was provided together with 12 reference conditions, mainly created according to the annex of
[i.1], table 1 shows one possible arrangement. Although it was observed that not all reference sets included exactly the
same speech material, used background noise, SNR ranges and speech distortion configuration, this data indicates
which range of speech and noise degradations can be expected in the databases.
For transforming the different databases (to achieve at least approximately on a common base for the retraining of the
model), thus the 12 x 3 values of the reference conditions (averaged over all samples) were used to linearly tr
...