SIST-V ETSI/EG 202 396-3 V1.5.1:2016

SLOVENSKI STANDARD
SIST-V ETSI/EG 202 396-3 V1.5.1:2016
01-februar-2016
.DNRYRVWSUHQRVDJRYRUDLQYHþSUHGVWDYQLKYVHELQ674=PRJOMLYRVWNDNRYRVWL
JRYRUDYSULVRWQRVWLãXPDR]DGMDGHO3UHQRVãXPDR]DGMD2EMHNWLYQH
SUHVNXVQHPHWRGH

Speech and multimedia Transmission Quality (STQ) - Speech Quality performance in the

presence of background noise - Part 3: Background noise transmission - Objective test

2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

The present document may be made available in electronic versions and/or in print. The content of any electronic and/or

print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any

existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the

print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.

Users of the present document should be aware that the document may be subject to revision or change of status.

Information on the current status of this and other ETSI documents is available at

If you find errors in the present document, please send your comment to one of the following services:

No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying

The content of the PDF version shall not be modified without the written authorization of ETSI.

DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.

3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and

GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.

Intellectual Property Rights ................................................................................................................................ 5

Foreword ............................................................................................................................................................. 5

Modal verbs terminology .................................................................................................................................... 5

1 Scope ........................................................................................................................................................ 6

2 References ................................................................................................................................................ 6

2.1 Normative references ......................................................................................................................................... 6

2.2 Informative references ........................................................................................................................................ 7

3 Symbols and abbreviations ....................................................................................................................... 8

3.1 Symbols .............................................................................................................................................................. 8

3.2 Abbreviations ..................................................................................................................................................... 8

4 Speech signals to be used ......................................................................................................................... 9

5 Selection of the data within the scope of the wideband objective model: Experts evaluation ............... 10

5.1 Selection process .............................................................................................................................................. 10

5.2 Results .............................................................................................................................................................. 10

5.3 French database ................................................................................................................................................ 11

6 Description of the wideband objective test method ............................................................................... 11

6.1 Introduction ...................................................................................................................................................... 11

6.2 Speech sample preparation and nomenclature .................................................................................................. 12

6.2.1 Speech sample preparation ......................................................................................................................... 12

6.2.2 Nomenclature .............................................................................................................................................. 14

6.3 Additional Training data .................................................................................................................................. 15

6.4 Principles of Relative Approach and Δ Relative Approach .............................................................................. 15

6.5 Objective N-MOS............................................................................................................................................. 19

6.5.1 Introduction................................................................................................................................................. 19

6.5.2 Description of N-MOS algorithm ............................................................................................................... 19

6.5.3 Comparing subjective and objective N-MOS results .................................................................................. 22

6.6 Objective S-MOS ............................................................................................................................................. 23

6.6.1 Introduction................................................................................................................................................. 23

6.6.2 Description of S-MOS Algorithm ............................................................................................................... 24

6.6.3 Comparing Subjective and Objective S-MOS Results ................................................................................ 27

6.7 Objective G-MOS............................................................................................................................................. 28

6.7.1 Description of G-MOS Algorithm .............................................................................................................. 28

6.7.2 Comparing subjective and objective G-MOS results .................................................................................. 29

7 Validation of the Wideband Objective Test Method .............................................................................. 30

7.1 Introduction ...................................................................................................................................................... 30

7.2 ETSI EG 202 396-2 Database Results Analysis ............................................................................................... 32

7.2.1 Comparing subjective and objective N-MOS results .................................................................................. 32

7.2.2 Comparing subjective and objective S-MOS results .................................................................................. 32

7.2.3 Comparing Subjective and Objective G-MOS Results ............................................................................... 33

7.3 Orange Validation Database results Analysed ................................................................................................. 34

7.3.0 Introduction................................................................................................................................................. 34

7.3.1 Comparing subjective and objective N-MOS results .................................................................................. 34

7.3.2 Comparing subjective and objective S-MOS results .................................................................................. 35

7.3.3 Comparing Subjective and Objective G-MOS Results ............................................................................... 35

8 Objective Model for Narrowband Applications ..................................................................................... 36

8.0 Introduction ...................................................................................................................................................... 36

8.1 File pre-processing ........................................................................................................................................... 36

8.2 Adaptation of the Calculations ......................................................................................................................... 37

8.3 Prediction results .............................................................................................................................................. 38

Annex A: Detailed post evaluation of listening test results ................................................................ 40

Annex B: Results of PESQ and TOSQA2001 - Analysis of ETSI EG 202 396-2 database .............. 43

Annex C: Comparison of objective MOS versus auditory MOS for the complete STF 294

database ................................................................................................................................. 50

Annex D: Comparison of objective MOS versus auditory MOS for rejected conditions ................ 52

Annex E: Void ........................................................................................................................................ 54

Annex F: Detailed STF 294 subjective and objective validation test results .................................... 55

Annex G: Void ........................................................................................................................................ 58

Training and Validation ....................................................................................................... 59

Annex I: Void ........................................................................................................................................ 61

Annex J: Summary of Czech samples not used for model training .................................................. 62

J.0 Introduction ...................................................................................................................................................... 62

J.1 Selection process - Czech database ........................................................................................................ 62

J.2 General differences between the databases ............................................................................................ 64

J.3 Comparison of the objective method results for Czech and French samples ......................................... 67

J.4 Czech conditions results analysis ........................................................................................................... 72

J.4.1 Comparing subjective and objective N-MOS results ....................................................................................... 72

J.4.2 Comparing subjective and objective S-MOS results ........................................................................................ 72

J.4.3 Comparing Subjective and Objective G-MOS Results..................................................................................... 73

J.5 Language Dependent Robustness of G-MOS......................................................................................... 74

J.6 Regression Coefficients for Czech data ................................................................................................. 75

J.7 Post selection .......................................................................................................................................... 76

Annex K: Relative Approach Non-Linear Transformation ............................................................... 80

Annex L: Bibliography .......................................................................................................................... 81

History .............................................................................................................................................................. 82

IPRs essential or potentially essential to the present document may have been declared to ETSI. The information

pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found

in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in

respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee

can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web

This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality

The present document is a deliverable of ETSI Specialized Task Force (STF) 294 entitled: "Improving the quality of

eEurope wideband speech applications by developing a performance testing and evaluation methodology for

The present document is part 3 of a multi-part deliverable covering Speech and multimedia Transmission Quality

(STQ); Speech Quality performance in the presence of background noise, as identified below:

Part 2: "Background noise transmission - Network simulation - Subjective test database and results";

In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and

"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of

"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

The present document aims to identify and define testing methodologies which can be used to objectively evaluate the

performance of narrowband and wideband terminals and systems for speech communication in the presence of

Background noise is a problem in mostly all situations and conditions and need to be taken into account in both,

terminals and networks. The present document provides information about the testing methods applicable to objectively

evaluate the speech quality in the presence of background noise. The present document includes:

• The description of the experts post evaluation process chosen to select the subjective test data being within the

• The results of the performance evaluation of the currently existing methods described in Recommendations

ITU-T P.862 [i.16] and P.862.1 [i.17] and in TOSQA2001 [i.19] which is chosen for the evaluation of

terminals in the framework of ETSI VoIP speech quality test events [i.8], [i.9], [i.10] and [i.11].

• The method which is applicable to objectively determine the different parameters influencing the speech

- ETSI ES 202 396-1 [i.1] which describes a recording and reproduction setup for realistic simulation of

background noise scenarios in lab-type environments for the performance evaluation of terminals and

- ETSI EG 202 396-2 [i.2] which describes the simulation of network impairments and how to simulate

realistic transmission network scenarios and which contains the methodology and results of the

- French speech sentences as defined in Recommendation ITU-T P.501 [i.13] for wideband and English

References are either specific (identified by date of publication and/or edition number or version number) or

non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the

Referenced documents which are not found to be publicly available in the expected location might be found at

NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee

The following referenced documents are necessary for the application of the present document.

References are either specific (identified by date of publication and/or edition number or version number) or

non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the

NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee

The following referenced documents are not necessary for the application of the present document but they assist the

[i.1] ETSI ES 202 396-1: "Speech and multimedia Transmission Quality (STQ); Speech quality

performance in the presence of background noise; Part 1: Background noise simulation technique

[i.2] ETSI EG 202 396-2: "Speech Processing, Transmission and Quality Aspects (STQ); Speech

Quality performance in the presence of background noise; Part 2: Background Noise Transmission

[i.3] Recommendation ITU-T P.835: "Subjective test methodology for evaluating speech

[i.4] Recommendation ITU-T P.800: "Methods for subjective determination of transmission quality".

[i.5] Recommendation ITU-T P.831: "Subjective performance evaluation of network echo cancellers".

[i.6] Genuit, K.: "Objective Evaluation of Acoustic Quality Based on a Relative Approach",

[i.7] Recommendation ITU-T SG 12 Contribution 34: "Evaluation of the quality of background noise

[i.8] ETSI 2 Speech Quality Test Event: "Anonymized Test Report", ETSI Plugtests, HEAD

NOTE: Available at: http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx. Also available as ETSI

NOTE: Available at: http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.

[i.11] ETSI 4 Speech Quality Test Event: "Anonymized Test Report "IP Gateways and IP Phones".

NOTE: Available at: http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.

[i.12] F. Kettler, H.W. Gierlich, F. Rosenberger: "Application of the Relative Approach to Optimize

[i.14] R. Sottek, K. Genuit: "Models of Signal Processing in human hearing", International Journal of

[i.15] SAE International - Document 2005-01-2513: "Tools and Methods for Product Sound Design of

[i.16] Recommendation ITU-T P.862: "Perceptual evaluation of speech quality (PESQ): An objective

method for end-to-end speech quality assessment of narrowband telephone networks and speech

[i.17] Recommendation ITU-T P.862.1: "Mapping function for transforming P.862 raw result scores to

[i.18] Recommendation ITU-T P.862.2: "Wideband extension to Recommendation P.862 for the

[i.19] Recommendation ITU-T SG 12 Contribution 19: "Results of objective speech quality assessment

[i.21] Recommendation ITU-T G.722.2: "Wideband coding of speech at around 16 kbit/s using Adaptive

[i.22] Recommendation ITU-T P.56: "Objective measurement of active speech level".

[i.26] M. Kendall: "Rank correlation methods", Charles Griffin & Company Limited, 1948.

[i.27] Sottek, R.: "Modelle zur Signalverarbeitung im menschlichen Gehör", PHD thesis RWTH Aachen,

[i.28] Recommendation ITU-T P.830: "Subjective performance assessment of telephone-band and

[i.30] ANSI S1.1-1986 (ASA 65-1986): "Specifications for Octave-Band and Fractional-Octave-Band

[i.31] Recommendation ITU-T G.160 Appendix II, Amendment 2: "Voice enhancement devices:

Revised Appendix II - Objective measures for the characterization of the basic functioning of

[i.32] ETSI TS 103 106: "Speech and multimedia Transmission Quality (STQ); Speech quality

performance in the presence of background noise: Background noise transmission for mobile

[i.33] Hastie, T.; Tibshirani, R.; Friedman, J.: "The Elements of Statistical Learning: Data Mining,

As with any objective model, the prediction of speech quality depends on the conditions under which the model was

tested and validated (see clauses 6.1 and 8). This dependency also applies to the speech material used in conjunction

The wideband version of the model uses French speech sentences. The near end speech signal (clean speech signal)

consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can be

The narrowband version of the model uses English speech sentences. The near end speech signal (clean speech signal)

consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can be

The aim of the selection process was to identify those data in the databases described in ETSI EG 202 396-2 [i.2] which

are consistent with the scope of the objective models to be studied within the present document.

The experts were selected on the based on the definition found in e.g. Recommendation ITU-T P.831 [i.5]: experts are

experienced in subjective testing. Experts are able to describe an auditory event in detail and are able to separate

different events based on specific impairments. They are able to describe their subjective impressions in detail. They

have a background in technical implementations of noise reduction systems and transmission impairments and do have

detailed knowledge of the influence of particular implementations on subjective quality.

Their task was to select the relevant conditions within the scope of the model to be developed. Therefore they had to

verify the consistency of the data with respect to the following selection criteria:

1) Artefacts others than the ones which should have been produced by the signal processing described in ETSI

EG 202 396-2 [i.2] e.g. due to the additional amplification required in order to provide a listening level of

2) Inconsistencies within one condition due to the selection of the individual speech samples from the database

3) Inconsistencies within one condition due to statistical variation of the signal processing described in ETSI

4) Inconsistencies due to Recommendation ITU-T P.56 [i.22] level adjustment process chosen for the complete