ETSI TR 103 503 V1.1.1 (2018-03)

Speech and multimedia Transmission Quality (STQ); Procedures for Multimedia Transmission Quality Testing with Parallel Task including Subjective Testing

DTR/STQ-255

General Information

• Status: Published
• Publication Date: 08-Mar-2018
• Technical Committee: STQ - Speech and multimedia Transmission Quality
• Current Stage: 12 - Completion
• Due Date: 22-Mar-2018
• Completion Date: 09-Mar-2018

Standards Content (Sample)

ETSI TR 103 503 V1.1.1 (2018-03)






TECHNICAL REPORT
Speech and multimedia Transmission Quality (STQ);
Procedures for Multimedia Transmission Quality Testing
with Parallel Task including Subjective Testing

---------------------- Page: 1 ----------------------
2 ETSI TR 103 503 V1.1.1 (2018-03)



Reference
DTR/STQ-255
Keywords
assessment, listening quality, parallel task

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2018.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M logo is protected for the benefit of its Members.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI TR 103 503 V1.1.1 (2018-03)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
Introduction . 4
1 Scope . 5
2 References . 5
2.1 Normative references . 5
2.2 Informative references . 5
3 Abbreviations . 6
4 Subjective speech quality assessment, intelligibility and listening effort: existing approaches . 7
4.1 Introduction . 7
4.2 Classification of parallel tasks in scientific publications . 7
4.2.1 Current approaches . 7
4.2.2 Mentally oriented tasks . 8
4.2.3 Physically oriented tasks . 8
4.2.4 Hybrid tasks . 9
5 Procedures for subjective testing deploying parallel task . 9
5.1 General considerations . 9
5.1.1 Introduction. 9
5.1.2 Task Class 1 (activity driven) . 9
5.1.3 Task Class 2 (purpose driven). 10
5.1.4 Additional comments to Task Class 1 and Task Class 2 classification . 10
5.2 Test Environment . 10
5.2.1 Real environment . 10
5.2.2 Lab with simulated parallel task . 10
5.2.3 VR based testing . 11
5.3 Subjective testing procedure. 11
5.4 Result Analysis and Reporting . 11
5.4.1 Introduction. 11
5.4.2 Special reported items . 11
Annex A: Examples of test scenarios incorporating parallel task . 12
A.1 Example scenario 1 - psychomotor experiment A (hybrid general task) . 12
A.2 Example scenario 2 - psychomotor experiment B (hybrid general task) . 13
A.3 Example scenario 3 - tasting experiment A (hybrid purpose oriented task). 14
A.4 Example scenario 4 - tasting experiment B (hybrid purpose oriented task) . 14
A.5 Example scenario 5 – stationary bicycle (physically oriented, purpose oriented task) . 15
A.6 Example scenario 6 - virtual reality deployment (physically oriented general task) . 15
History . 17


ETSI

---------------------- Page: 3 ----------------------
4 ETSI TR 103 503 V1.1.1 (2018-03)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Speech and multimedia Transmission
Quality (STQ).
The present document describes auditory test methodologies for the prediction of perceived audio signal quality under
parallel task conditions.
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
Subjective testing of speech quality and intelligibility is standardized at ETSI, ANSI, ITU-T and ITU-R. Tests are
performed in defined environments using listening/conversational rigorous procedures (Recommendation
ITU-T P.800 [i.16], Recommendation ITU-T P.805 [i.21], Recommendation ITU-T P.835 [i.18], Recommendation
ITU-R BS.1534-3 [i.22], Recommendation ITU-R BS.1116 [i.23], etc.), and they require relaxed, fresh, fit and
concentrated naive or expert listeners seated comfortably in usually artificially looking listening room/booth.
However, such a test does not correspond to the normal use of the tested technologies. Voice services are often used in
sports, driving, work, public transport, or other noisy or less convenient environments. Users are tired, stressed or
concentrate on another, often important, task.
In an attempt to bring laboratory tests closer to reality, the so-called dual-task or parallel-task tests are introduced, in
these test participants are asked to perform multiple different tasks at the same time.

ETSI

---------------------- Page: 4 ----------------------
5 ETSI TR 103 503 V1.1.1 (2018-03)
1 Scope
The present document describes the methods for assessment of subjective audio (including speech) quality and speech
intelligibility under parallel task condition. This approach can be used to evaluate the perceived listening quality or
speech intelligibility in situations which better mimics real operation of the tested telecommunication equipment or
algorithm.
The present document describes possible parallel task generation and scenarios, the test design and reference conditions
used to evaluate the quality or intelligibility subjectively.
Several parallel task scenarios are covered:
• Physically oriented.
• Mentally oriented.
• Hybrid.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] V. Durin and L. Gros: "Measuring speech quality impact on tasks performance", Proc. Annu.
Conf. Int. Speech Commun. Assoc. INTERSPEECH, pp. 2074-2077, 2008.
[i.2] A. Serampalis, S. Kalluri, B. Edwards, and E. Hafter: "Objective measures of listening effort in
noise", J. Speech, Lang. Hear. Res., vol. 52, no. October 2009, pp. 1230-1240, 2009.
[i.3] G. P. Sonntag, T. Portele, and F. Haas: "Comparing the comprehensibility of different synthetic
voices in a dual task experiment", Proc. Third Work. Speech Synth. Jenolan Caves House, Blue
Mt., pp. 5-10, 1998.
[i.4] L. Gros, N. Chateau, and S. Busson: "The impact of real environments on transmitted speech
quality judgments", Quality, vol. 0, pp. 45-50, 2003.
[i.5] D. Guse, S. Egger, A. Raake, and S. Moller: "Web-QOE under real-world distractions: Two test
cases", 2014 6th Int. Work. Qual. Multimed. Exp. QoMEX 2014, pp. 220-225, 2014.
[i.6] S. L. Beilock, T. H. Carr, C. MacMahon, and J. L. Starkes: "When paying attention becomes
counterproductive: Impact of divided versus skill-focused attention on novice and experienced
performance of sensorimotor skills", J. Exp. Psychol. Appl., vol. 8, no. 1, pp. 6-16, 2002.
[i.7] J. Holub: "Low Bit-rate Coded Speech Intelligibility - Comparison of Laboratory Test Results and
Results of Test with Parallel Task", in Future Forces Forum, 2016.
ETSI

---------------------- Page: 5 ----------------------
6 ETSI TR 103 503 V1.1.1 (2018-03)
[i.8] D. L. Strayer and W. A. Johnston: "Driven to distraction: Dual-task studies of simulated driving
and conversing on a cellular telephone", Psychol. Sci., vol. 12, no. 6, pp. 462-466, 2001.
[i.9] S. Choi, A. Lotto, D. Lewis, B. Hoover, and P. Stelmachowicz: "Attentional Modulation of Word
Recognition by Children in a Dual-Task Paradigm", J. Speech Lang. Hear. Res., vol. 51, no. 4,
p. 1042, Aug. 2008.
[i.10] Y.-H. Wu, E. Stangl, X. Zhang, J. Perkins, and E. Eilers: "Psychometric Functions of Dual-Task
Paradigms for Measuring Listening Effort", Ear Hear., vol. 37, no. 6, pp. 660-670, 2016.
[i.11] C. Kwak and W. Han: "Comparison of Single-Task versus Dual-Task for Listening Effort",
J. Audiol. Otol., Oct. 2017.
[i.12] L. Gros, N. Chateau, and A. Macé: "Assessing speech quality : a new approach Methodology",
2005.
[i.13] K. S. Helfer, J. Chevalier, and R. L. Freyman: "Aging, spatial cues, and single- versus dual-task
performance in competing speech perception", J. Acoust. Soc. Am., vol. 128, no. 6,
pp. 3625-3633, Dec. 2010.
[i.14] K. Bunton and C. K. Keintz: "The use of a dual-task paradigm for assessing speech intelligibility
in clients with Parkinson disease", J. Med. Speech. Lang. Pathol., vol. 16, no. 3, pp. 141-155,
Sep. 2008.
[i.15] ITU-T Handbook: "Practical procedures for subjective testing", 2011.
[i.16] Recommendation ITU-T P.800 (08/1996): "Methods for subjective determination of transmission
quality".
[i.17] Recommendation ITU-T P.807 (02/2016): "Subjective test methodology for assessing speech
intelligibility".
[i.18] Recommendation ITU-T P.835 (11/2003): "Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[i.19] Council of Europe (2011): "Common European Framework of Reference for Languages: Learning,
Teaching, Assessment" Council of Europe.
[i.20] Recommendation ITU-T P.1400 (03/2013): "Statistical analysis, evaluation and reporting
guidelines of quality measurements".
[i.21] Recommendation ITU-T: P.805: "Subjective evaluation of conversational quality", Geneva 2007.
[i.22] Recommendation ITU-R: BS.1534: "Method for the subjective assessment of intermediate quality
levels of coding systems", Geneva 2015.
[i.23] Recommendation ITU-R: BS.1116: "Methods for the subjective assessment of small impairments
in audio systems", Geneva 2015.
3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ECG ElectroCardioGraphy
EEG ElectroEncephaloGraphy
PC Personal Computer
QoE Quality of Experience
STD STandard Deviation
VR Virtual Reality
ETSI

---------------------- Page: 6 ----------------------
7 ETSI TR 103 503 V1.1.1 (2018-03)
4 Subjective speech quality assessment, intelligibility
and listening effort: existing approaches
4.1 Introduction
Subjective testing of speech quality and intelligibility follows strictly standardized procedures. Tests are performed in
defined environments using listening/conversational rigorous procedures ([i.16], [i.18], etc.) and it requires relaxed,
fresh, fit and concentrated naive or expert listeners comfortably seated in a listening room/booth with proper acoustic
lining to minimize e.g. inherent background noise and room reverberation.
However, such a test does not correspond to normal use of the tested technologies. Voice services are often used during
sports, driving, work, etc. Users are tired, stressed or concentrated on another, often important, task.
To bring laboratory tests closer to reality, the so-called dual-task or parallel-task tests are introduced, where test
participants are asked to perform multiple different tasks at the same time. The test results obtained during parallel task
test differ from regular subjective tests. The differences are sometimes contra-intuitive and cannot be explained e.g. by
decreased level of subjects' attention. The parallel task should be designed to distract subjects in a similar way as the
activity performed during the real (targeted) situation. Limitations are given by requirements on repeatability, space-
and movement- restrictions in the lab, etc.
4.2 Classification of parallel tasks in scientific publications
4.2.1 Current approaches
Parallel tasks found in scientific literature can be divided into three types: Mentally oriented tasks, Physically oriented
tasks, and Hybrid tasks. Selected available experiments of those three categories are discussed in Table 1.
Table 1: Resource summary
Reference Test type Parallel task Parallel task type Language
[i.1] Speech intelligibility Memorizing digits Mentally oriented N/A
[i.2] Speech intelligibility Memorizing digits Mentally oriented English
[i.3] Speech intelligibility Pressing colour buttons Mentally oriented German
[i.4] QoE test Pressing colour buttons Mentally oriented English
Traveling in public Mentally oriented;
[i.5] QoE test German
transport; watching a TV Hybrid
Memorizing tones;
[i.6] Other Mentally oriented N/A
memorizing words
[i.7] Speech intelligibility Laser shooting simulator Hybrid English
[i.8] Other Telephone call Hybrid English
Word repetition;
[i.9] Speech intelligibility Mentally oriented English
Memorizing digits
[i.10] Speech intelligibility Pressing colour button Mentally oriented English
Memorizing sentences,
[i.11] Speech intelligibility Mentally oriented Korean
Arithmetic
Matching coloured
[i.12] QoE test Mentally oriented N/A
squares
Forward ⁄backward
[i.13] Speech intelligibility discrimination and speech Mentally oriented English
understanding
[i.14] Speech intelligibility Turning a nut on a bolt Mentally oriented English

ETSI

---------------------- Page: 7 ----------------------
8 ETSI TR 103 503 V1.1.1 (2018-03)
4.2.2 Mentally oriented tasks
Frequently used mental tasks are memory-related tasks requiring memorization and subsequent repetition of
information, most often words or digits. In experiment [i.1], listeners had to identify the letter as prescribed, while
remembering the five digits displayed or played before this description. The results of the experiment depend on both
the quality of the codec used and the intelligibility of the description, and on the way the numbers are presented and
how the conditions are sorted (serial/random). A memory task is also used in other experiments, such as in [i.2], [i.9]
and [i.11]. In the first experiment [i.2], the primary test condition consisted in the different levels of noise in the
background of test sentences. The listeners had the task of repeating the last word of the sentence heard or trying to
guess it if it was not comprehensible. The second task of the listeners was to remember all the last words and repeat
them after eight sentences. In the next experiment [i.9], a group of 64 children participated in speech intelligibility test.
Half of them were told to pay their primary attention to word repetition and the other half to remember digits. Single-
task and dual-task performances were
...