ETSI TR 103 559 V1.1.1 (2019-08)

Speech and multimedia Transmission Quality (STQ); Best practices for robust network QoS benchmark testing and scoring

DTR/STQ-00219m

General Information

• Status: Published
• Publication Date: 21-Aug-2019
• Technical Committee: STQ MOBILE - STQ Mobile
• Current Stage: 12 - Completion
• Due Date: 22-Aug-2019
• Completion Date: 22-Aug-2019

Standards Content (Sample)

ETSI TR 103 559 V1.1.1 (2019-08)






TECHNICAL REPORT
Speech and multimedia Transmission Quality (STQ);
Best practices for robust network QoS
benchmark testing and scoring

---------------------- Page: 1 ----------------------
2 ETSI TR 103 559 V1.1.1 (2019-08)



Reference
DTR/STQ-00219m
Keywords
3G, benchmarking, data, GSM, LTE, network,
QoE, QoS, scoring, service, speech, video,
ViLTE, VoLTE
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2019.
All rights reserved.

DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.

3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI TR 103 559 V1.1.1 (2019-08)
Contents
Intellectual Property Rights . 6
Foreword . 6
Modal verbs terminology . 6
Introduction . 6
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definition of terms, symbols and abbreviations . 8
3.1 Terms . 8
3.2 Symbols . 8
3.3 Abbreviations . 8
4 Governing Principles for Mobile Benchmarking . 8
4.1 General . 8
4.2 Fair Play . 8
4.3 Comparing networks with different coverage extents . 9
4.4 Comparing networks with differing technology use . 9
4.5 Test device selection . 9
4.6 Test server selection . 9
4.7 Test method transparency . 10
4.8 Advice and best practice for web-page selection . 10
5 General Description . 11
6 Test Areas . 12
6.1 General . 12
6.2 Geographical divisions . 12
6.2.1 Cities . 12
6.2.2 Roads . 12
6.2.3 Complementary areas . 13
7 User Profiles . 13
8 Test Metrics . 13
8.1 Introduction . 13
8.2 Telephony . 13
8.2.1 General . 13
8.2.2 Telephony Success Ratio . 13
8.2.3 Setup Time . 14
8.2.4 Listening Quality . 14
8.3 Video Testing . 14
8.3.1 General . 14
8.3.2 Video Streaming Service Success Ratio . 14
8.3.3 Setup Time . 14
8.3.4 Video Quality . 14
8.4 Data Testing . 14
8.4.1 General . 14
8.4.2 Success Ratio . 15
8.4.3 Throughput . 15
8.5 Services Testing . 15
8.5.1 General . 15
8.5.2 Services . 15
8.5.2.1 Browsing . 15
8.5.2.2 Social Media . 15
8.5.2.3 Messaging . 15
ETSI

---------------------- Page: 3 ----------------------
4 ETSI TR 103 559 V1.1.1 (2019-08)
8.5.3 Success Ratio . 15
8.5.4 Timings . 16
9 Weighting . 16
9.1 General . 16
9.2 Areas . 16
9.3 Tests . 17
9.3.1 General . 17
9.3.2 Telephony . 18
9.3.2.1 General . 18
9.3.2.2 Scoring . 18
9.3.3 Video streaming . 19
9.3.3.1 General . 19
9.3.3.2 Scoring . 19
9.3.4 Data Testing . 19
9.3.4.1 General . 19
9.3.4.2 Scoring . 20
9.3.5 Service Testing . 20
9.3.5.1 General . 20
9.3.5.2 Scoring . 20
10 Statistical confidence and robustness . 21
10.1 General . 21
10.2 Influence of the derived scores on statistical confidence . 21
10.3 Statistical confidence level estimation . 22
10.3.1 General . 22
10.3.2 Statistical analysis using a bootstrap resampling method . 22
10.3.3 Interpretation of results . 22
Annex A: Example set of weighting factors, limits and thresholds . 23
A.1 General . 23
A.2 Area . 23
A.2.1 Geographical divisions . 23
A.2.1.1 General . 23
A.2.1.2 City type . 23
A.2.1.3 Road type . 23
A.2.1.4 Complementary areas . 23
A.3 Mobile services . 24
A.4 Test metrics of mobile services . 24
A.4.1 General . 24
A.4.2 Telephony . 24
A.4.3 Data Services . 24
A.4.3.1 General . 24
A.4.3.2 Video Streaming . 24
A.4.3.3 Data Testing . 25
A.4.3.4 Browsing . 25
A.4.3.5 Social Media and Messaging . 25
A.5 Example Calculation . 26
Annex B: Example set of weighting factors, limits and thresholds . 27
B.1 General . 27
B.2 Area . 27
B.2.1 Geographical divisions . 27
B.3 Mobile services . 28
B.4 Test metrics of mobile services . 28
B.4.1 Telephony . 28
B.4.2 Data Services . 28
ETSI

---------------------- Page: 4 ----------------------
5 ETSI TR 103 559 V1.1.1 (2019-08)
B.4.2.1 General . 28
B.4.2.2 Video Streaming . 28
B.4.2.3 Data Testing . 29
B.4.2.3.1 File Download (based on 3 MB File Size) . 29
B.4.2.3.2 File Upload (based on 1 MB File Size) . 29
B.4.2.3.3 File Download (based on 7 s Fixed Download Time) . 29
B.4.2.3.4 File Upload (based on 7 s Fixed Upload Time) . 30
B.4.2.4 Browsing . 30
B.4.2.4.1 Static Web Page . 30
B.4.2.4.2 Dynamic Web Pages . 30
B.5 Remarks on mapping functions . 31
B.6 Example Calculation . 32
History . 33


ETSI

---------------------- Page: 5 ----------------------
6 ETSI TR 103 559 V1.1.1 (2019-08)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Speech and multimedia Transmission
Quality (STQ).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
Countrywide mobile network benchmarking and scoring campaigns published in the press enjoy great public interest
and are of high importance for the operators of mobile networks. A first place score in press releases associated with
such measurements is often used in the advertisements of the winning operator to boost their corporate identity. Though
published results are often well documented, they are not always completely transparent about how the actual scoring
has been achieved. Methods and underlying assumptions are mostly not described in detail.
The present document discusses the construction and methods of such a countrywide measurement campaign, with
respect to the area and population to be covered, the collection and aggregation of the test results and the weighting of
the various aspects tested. The applicability of the results of such a campaign, for inter country comparison purposes, is
not covered in the present document.
Based on established methods and quality metrics, such as success ratio and setup times, the results of the data collected
in the benchmarking are aggregated individually. The individual aggregated values are weighted and further aggregated
for each application like telephony, video and data services. The application fields are then in turn weighted and
aggregated over the different areas where the data is collected. Finally, calculation of an overall score or a joint score is
performed.
The experienced quality of service varies over time so that the individual score of a particular throughput cannot be
fixed once and for all. As well as the test metrics changing over time, so does the importance of the various services.
The present document describes a typical set of tests that could be performed and a related evaluation criteria. In the
annexes, actual real-world examples of weightings and score mapping parameters are given.
ETSI

---------------------- Page: 6 ----------------------
7 ETSI TR 103 559 V1.1.1 (2019-08)
1 Scope
The present document describes the best practices for benchmarking of mobile networks. The goal of the benchmarking
is to determine the best provider or operator for a designated area with respect of the services accessed with a mobile
phone. The tests conducted are telephony, video streaming, data throughput and more interactive applications such as
browsing, social media and messaging. This goal is achieved by executing benchmarking tests in designated test areas
that represent or actually cover a major part of the users of mobile services. The results collected in the various areas are
individually and collectively weighted and summarized into an overall score.
Due to the rapid development of the mobile technology and consumption habits of the users, the quality of experience
of the users changes over time even when the objective to measure the quality of service does not change. The present
document needs to keep up with those changes and does so by parameterizing the individual factors that contribute to
the score.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
[i.2] Void.
[i.3] ETSI TR 102 505: "Speech and multimedia Transmission Quality (STQ); Development of a
Reference Web page".
[i.4] ETSI TR 101 578: "Speech and multimedia Transmission Quality (STQ); QoS aspects of
TM
TCP-based video services like YouTube ".
[i.5] ETSI TR 102 678: "Speech and multimedia Transmission Quality (STQ); QoS Parameter
Measurements based on fixed Data Transfer Times".
[i.6] ETSI TR 103 138: "Speech and multimedia Transmission Quality (STQ); Speech samples and
their use for QoS testing".
[i.7] Recommendation ITU-T E.840: "Statistical framework for end-to-end network-performance
benchmark scoring and ranking".
[i.8] Recommendation ITU-T P.1401: "Methods, metrics and procedures for statistical evaluation,
qualification and comparison of objective quality prediction models".
[i.9] Recommendation ITU-T P.863: "Perceptual objective listening quality prediction".
[i.10] Recommendation ITU-T P.863.1: "Application guide for Recommendation ITU-T P.863".
ETSI

---------------------- Page: 7 ----------------------
8 ETSI TR 103 559 V1.1.1 (2019-08)
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
live web page: web pages considered as dynamic content, content changes over time and some content might be
different caused by the hosting server or the access network
static web page: web pages considered as static content, content stays constant over time and access network
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
AMR Adaptive Multi-Rate
API Application Programming Interface
CDN Content Delivery Network
CST Call Setup Time
DL DownLink
DNS Domain Name System
EVS Enhanced Voice Services
FB FullBand
HD High Definition
HTTP HyperText Transfer Protocol
IP Internet Protocol
ITU International Telecommunication Union
ITU-T International Telecommunication Union Telecommunication
KPI Key Performance Indicator
MB MegayByte
MOS Mean Opinion Sc
...