ETSI TS 103 222-2 V1.1.1 (2017-12)

Speech and multimedia Transmission Quality (STQ); Reference benchmarking, background traffic profiles and KPIs; Part 2: Reference benchmarking and KPIs for High speed internet

DTS/STQ-219-2

General Information

• Status: Published
• Publication Date: 17-Dec-2017
• Technical Committee: STQ - Speech and multimedia Transmission Quality
• Current Stage: 12 - Completion
• Due Date: 26-Dec-2017
• Completion Date: 18-Dec-2017

Standards Content (Sample)

ETSI TS 103 222-2 V1.1.1 (2017-12)






TECHNICAL SPECIFICATION
Speech and multimedia Transmission Quality (STQ);
Reference benchmarking,
background traffic profiles and KPIs;
Part 2: Reference benchmarking
and KPIs for High speed internet

---------------------- Page: 1 ----------------------
2 ETSI TS 103 222-2 V1.1.1 (2017-12)



Reference
DTS/STQ-219-2
Keywords
KPI, QoS

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2017.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M logo is protected for the benefit of its Members.
GSM® and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI TS 103 222-2 V1.1.1 (2017-12)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions and abbreviations . 6
3.1 Definitions . 6
3.2 Abbreviations . 7
4 Quality of service parameter values for High-speed internet . 7
4.0 KPIs overview . 7
4.1 PING . 12
4.1.1 PING Packets Missing [%] . 12
4.1.2 PING Packet Errors [%] . 13
4.1.3 PING Failure Ratio [%] . 13
4.2 HTTP . 13
4.2.1 HTTP Response Time . 13
4.2.2 HTTP Download Throughput . 13
4.2.3 HTTP Download Throughput [%] of Bandwidth . 14
4.2.4 HTTP Percentage of invoiced Bandwidth [%] . 14
4.2.5 HTTP Upload Response Time . 14
4.2.6 HTTP Upload Throughput . 14
4.2.7 HTTP Upload Throughput [%] of Bandwidth . 14
4.2.8 Number of HTTP Sessions . 14
4.2.9 HTTP Dropped Sessions [%] . 15
4.2.10 HTTP Download Failure Ratio [%] . 15
4.2.11 HTTP peak download throughput . 15
4.3 FTP . 15
4.3.1 FTP Upload/Download Throughput . 15
4.3.2 FTP Upload/Download Throughput [%] of Bandwidth . 15
4.3.3 FTP Percentage of invoiced Bandwidth [%] . 15
4.3.4 FTP Download Throughput [%] of Bandwidth . 15
4.3.5 File Download/Upload Data Transfer Cut-off [%] . 15
4.3.6 FTP {Download|Upload} Service non-accessibility [%] . 16
4.3.7 FTP {Download|Upload} Retransmission Ratio . 16
4.3.8 FTP {Download|Upload} packet loss rate [%] . 16
4.3.9 FTP Retransmission Ratio [%] . 16
4.4 TCP . 16
4.4.1 TCP Round Trip Time (Server side) . 16
4.4.2 TCP Round Trip Time (Client side) . 17
4.4.3 TCP Retransmission Ratio . 17
4.4.4 TCP Data Call Access Failure Ratio . 17
4.4.5 TCP Data Call Access Time . 17
4.4.6 TCP Server Response Time . 18
5 Cloud Services . 18
5.0 Introduction . 18
5.1 DNS Lookup Time . 18
5.2 Web browsing indicators . 18
5.2.1 Introduction. 18
5.2.2 HTTP Concepts . 19
5.2.3 Website Response Time . 20
ETSI

---------------------- Page: 3 ----------------------
4 ETSI TS 103 222-2 V1.1.1 (2017-12)
5.2.4 Website Load Duration Time. 20
5.2.5 Website Session Duration Time. 21
5.2.6 Website Download Failure Ratio . 22
6 Streaming . 23
6.0 QoE Streaming Indicators . 23
6.1 Streaming indicators . 23
6.2 Streaming Reproduction Cut-Off Ratio . 24
6.3 Time to Stream Start. 24
6.4 Streaming Reproduction Start Failure Ratio [%] . 25
6.5 Total Number of Videos . 25
6.6 Total Reproduction Time . 25
6.7 Effective Reproduction Time . 25
6.8 Rebuffering Time Percentage . 25
6.9 Streaming Mean Data Rate . 26
6.10 Streaming Peak Data Rate . 26
6.11 Effective Reproduction Time . 26
6.12 Number Streaming Sessions . 26
7 Gaming . 26
7.0 Introduction . 26
7.1 Number of Gaming Sessions . 27
7.2 Gaming Session Duration . 27
7.3 Gaming Average Throughput Uplink . 27
7.4 Gaming Average Throughput Downlink . 27
7.5 Gaming Packet Loss Ratio (Uplink/Downlink) . 27
7.6 Gaming TCP Retransmission Ratio (Uplink/Downlink) . 28
7.7 Gaming End-to-end Latency . 28
7.8 Gaming Traffic Volume (Uplink/Downlink) . 29
7.9 Daily Number of Unique Users . 29
History . 30


ETSI

---------------------- Page: 4 ----------------------
5 ETSI TS 103 222-2 V1.1.1 (2017-12)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Specification (TS) has been produced by ETSI Technical Committee Speech and multimedia
Transmission Quality (STQ).
The present document is part 2 of a multi-part deliverable covering Reference benchmarking, background traffic
profiles and KPIs as identified below:
Part 1: "Reference benchmarking, background traffic profiles and KPIs for VoIP and FoIP in fixed networks";
Part 2: "Reference benchmarking and KPIs for High speed internet";
Part 3: "Reference benchmarking, background traffic profiles and KPIs for UMTS and VoLTE";
Part 4: "Reference benchmarking for IPTV, Web TV and RCS-e Video Share".
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
The present document describes the reference benchmarking, background traffic profiles and key performance
indicators for high-speed internet.

ETSI

---------------------- Page: 5 ----------------------
6 ETSI TS 103 222-2 V1.1.1 (2017-12)
1 Scope
The offer of new NGN services requires new KPIs, QoS measurement and benchmarking methods which are needed to
ensure the quality of new services. To ensure the comparability of test results, reference benchmarking methods and
background traffic load profiles are needed. The present document describes key performance indicators and
benchmarking methods for the spectrum of potential applications. All access technologies offered by the operator under
test are considered.
The present document is the second part of the multi-part deliverable which consists of four parts. The present
document contains the Reference benchmarking and KPIs for High speed internet.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] Recommendation ITU-T E.800 (2008): "Definitions of terms related to quality of service".
[2] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI TR 101 578: "Speech and multimedia Transmission Quality (STQ); QoS aspects of TCP-
TM
based video services like YouTube ".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following definition applies:
benchmark: evaluation of performance value/s of a parameter or set of parameters for the purpose of establishing
value/s as the norm against which future performance achievements may be compared or assessed
ETSI

---------------------- Page: 6 ----------------------
7 ETSI TS 103 222-2 V1.1.1 (2017-12)
NOTE: The definition is taken from Recommendation ITU-T E.800 [1].
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ACK Acknowledgement message
DL DownLoad
DNS Domain Name System
E2E End-to-End
FFS For Further Study
FIN FIN message
FTP File Transfer Protocol
GET HTTP method GET
HTTP Hypertext Transfer Protocol
IAD Integrated Access Device
ICMP Internet Control Message Protocol
IP Internet Protocol
IPDV IP Delay Variation
IPER IP Packet Error Ratio
IPLR IP Packet Loss Ratio
IPTD IP Packet Transfer Delay
IPTV Internet Protocol Television
KPI Key Performance Indicator
NGN Next Generation Networks
PING Packet INternet Groper (Send a packet to a computer and wait for its return)
POST HTTP method POST
QoE Quality of Experience
QoS Quality of Service
RTP Real-Time Transport Protocol
RTT Round-Trip Time
SYN SYNchronize Message
TCP Transmission Control Protocol
UDP User Datagram Protocol
UE User Equipment
UL UpLoad
UMTS Universal Mobile Telecommunications System
URI Uniform Resource Identifier
VoIP Voice over IP
4 Quality of service parameter values for High-speed
internet
4.0 KPIs overview
To determine the quality of high-speed internet connections, the following measurement and KPI values are determined.
Table 4.0-1: Overview of quality benchmarks for DNS
1. DNS Lookup Time [ms]; see clause 5.1
2. DNS Lookup Failure Ratio [%]; see clause 5.10 of [2]
3. DNS Host Name Resolution Time; see clause 5.11 of [2]

ETSI

---------------------- Page: 7 ----------------------
8 ETSI TS 103 222-2 V1.1.1 (2017-12)
Table 4.0-2: Overview of quality benchmarks for PING
1. PING Average Time [ms]; is defined in clause 6.3.1 of [2]
2. PING packets missing [%]; see clause 4.1.1
3. PING Packets Errors [%]; see clause 4.1.2
4. PING Failure Ratio; see clause 4.1.3

Table 4.0-3: Overview of quality benchmarks for HTTP
1. HTTP Service Non-Accessibility [%]; see clause 6.8.1 of [2]
2. HTTP Setup Time [s]; see clause 6.8.2 of [2]
3. HTTP IP Service Access Failure Ratio [%]; see clause 6.8.3 of [2]
4. HTTP IP Service Setup Time [s]; see clause 6.8.4 of [2]
5. HTTP Session Failure Ratio [%]; see clause 6.8.5 of [2]
6. HTTP Session Time [s]; see clause 6.8.6 of [2]
7. HTTP Mean Data Rate [kbit/s]; see clause 6.8.7 of [2]
8. HTTP Data Transfer Cut-off Ratio [%]; see clause 6.8.8 of [2]
9. HTTP Content Compression Ratio [%]; see clause 6.8.9 of [2]
10. HTTP peak download throughput [kbit/s]; see clause 4.2.11
11. HTTP Response Time; see clause 4.2.1
12. HTTP Download Throughput; see clause 4.2.2
13. HTTP Download Throughput < x % of Bandwidth; see clause 4.2.3
14. HTTP Percentage of invoiced Bandwidth; see clause 4.2.4
15. HTTP Upload Response Time; see clause 4.5
16. HTTP HTTP Upload Throughput; see clause 4.2.6
17. HTTP HTTP Upload Throughput; see clause 4.2.7
18. HTTP; Number of HTTP Sessions; see clause 4.2.8
19. HTTP HTTP Dropped Sessions; see clause 4.2.9
22. Number of HTTP Sessions (Daily number of http sessions); see clause 4.2.9
23. Traffic Volume DL (Daily total volume in Downlink)
24. Traffic Volume UL (Daily total volume in Uplink)
25. Number of active users
26. Average duration of HTTP sessions [min]

ETSI

---------------------- Page: 8 ----------------------
9 ETSI TS 103 222-2 V1.1.1 (2017-12)
Table 4.0-4: Overview of quality of service parameters for FTP
1. FTP {Download|Upload} Setup Time [s]; see clause 6.1.2 of [2]
2. FTP {Download|Upload} IP-Service Access Failure Ratio [%]; see clause 6.1.3 of [2]
3. FTP {Download|Upload} IP-Service Setup Time [s]; see clause 6.1.4 of [2]
4. FTP {Download|Upload} Session Failure Ratio [%]; see clause 6.1.5 of [2]
5. FTP {Download|Upload} Session Time [s]; see clause 6.1.6 of [2]
6. FTP {Download|Upload} Mean Data Rate [kbit/s] ]; see clause 6.1.7 of [2]
7. FTP {Download|Upload} Data Transfer Cut-off Ratio [%]; see clause 4.3.7
8. FTP {Download|Upload} Service non - Accessibility [%]; see clause 4.3.8
9. FTP {Download|Upload} throughput [kbit/s]; defined in clause 4.3.1
10. FTP Upload/Download Throughput < x % of Bandwidth; defined in clause 4.3.2
11. FTP Percentage of invoiced Bandwidth; defined in clause 4.3.3
12. FTP Download Throughput < x % of Bandwidth; defined in clause 4.3.4
13. File Download/Upload Data Transfer Cut-off; defined in clause 4.3.5
14. FTP {Download|Upload} Service non - Accessibility [%]; defined in clause 4.3.6
15. FTP Retransmission Ratio; defined in clause 4.3.9
16. FTP {Download|Upload} packet loss rate [%]; defined in clause 4.3.8
17. FTP {Download|Upload} Traffic Volume (Daily total volume in Downlink)
18. Active Users (Daily number of unique users)
19. FTP upload with parallel HTTP Download
20. FTP upload Number of FTP Sessions with parallel HTTP Download
21. FTP upload Service non - Accessibility with parallel HTTP Download [%]
22. FTP upload Setup Time with parallel HTTP Download
23. FTP upload IP service Access Failure Ratio with parallel HTTP Download [%]
24. FTP upload IP-Service Access Setup Time with parallel HTTP Download [%]
25. FTP upload Session Time [s] with parallel HTTP Download
26. FTP upload Mean Data Rate with parallel HTTP Download [kbit/s]
27. FTP upload Data Transfer Cut-off Ratio with parallel HTTP Download [%]

Table 4.0-5: Overview of TCP quality benchmarks
1. TCP Round Trip Time (Server side); see clause 4.4.1
2. TCP Round Trip Time (Client side); see clause 4.4.2
3. TCP Retransmission Ratio; see clause 4.4.3
4. TCP Data Call Access Failure Ratio; see clause 4.4.4
5. TCP Data Call Access Time; see clause 4.4.5
6. TCP Server Response Time; see clause 4.4.6

Table 4.0-6: Overview of quality benchmarks E-mail
1. E-Mail {Download|Upload} Service Non-Accessibility [%]; see clause 7.2 of [2]
2. E-Mail {Download|Upload} Setup Time [s]; see clause 7.2 of [2]
3. E-Mail {Download|Upload} IP-Service Access Failure Ratio [%]; see clause 7.2 of [2]
4. E-Mail {Download|Upload} IP-Service Setup Time [s]; see clause 7.2 of [2]
5. E-mail {Upload|Download} Session Failure Ratio [%]; see clause 7.2 of [2]
6. E-mail {Upload|Header Download|Download} Session Time [s]; see clause 7.2 of [2]
7. E-mail {Upload|Header Download|Download} Mean Data Rate [kbit/s]; see clause 7.2 of [2]
8. E-mail {Upload|Header Download|Download} Data Transfer Cut-off Ratio [%];
see clause 7.2 of [2]
9. E-mail {Upload|Header Download|Download} Data Transfer Time [s]; see clause 7.2 of [2]
10. E-mail Login Non-Accessibility [%]; see clause 7.2 of [2]
11. E-mail Login Access Time [s]; see clause 7.2 of [2]
12. E-mail Notification Push Failure Ratio [%]; see clause 7.2 of [2]
13. E-mail Notification Push Transfer Time [s]; see clause 7.2 of [2]
14. E-mail End-to-End Failure Ratio [%]; see clause 7.2 of [2]

ETSI

---------------------- Page: 9 ----------------------
10 ETSI TS 103 222-2 V1.1.1 (2017-12)
Table 4.0-7: Network Diagnostic parameters
1. Number of open parallel UDP ports (Default value 11)
2. Number of open parallel TCP ports (Default value 16)
3. Reachability of DNS entries (Default value 45)
4. Reachability of Transparent connections (Default value 2) - for further study
5. Reachability of Reference Webpage (Default value 1) - for further study
6. Unchanged content transport (Default value 2) - for further study
7. Radio level - for further study
8. Stability of Radio level - for further study

Table 4.0-8: Overview of quality benchmarks for Network performance objectives for IP-based
services
1. Maximal IPDV
2. Maximal IPTD
3. Maximal IPLR
4. Maximal IPER

Table 4.0-9: Overview of quality benchmarks for voice Media RTP
1. RTP DL Throughput
2. RTP UL Throughput
3. RTP Latency
4. RTP Jitter
5. RTP Out of sequence
6. RTP Dropped sequence
7. RTP duplicate packets

Table 4.0-10: Overview of quality benchmarks for real-time video
1. DL Throughput
2. UL Throughput
3. Latency
4. Jitter
5. Out of sequence
6. Dropped sequence
7. Duplicate packets

Table 4.0-11: Web Browsing QoS Parameters
1. Website Response time; see clause 5.2.3
2. Website Load Duration Time; see clause 5.2.4
3. Website load duration; see clause 5.2.5
4. Website session duration; see clause 5.2.6
5. Website download Failure ratio; see clause 5.2.7
6. HTTP Service Non-Accessibility; defined in clause 6.8.1 of [2]
7. Number HTTP Sessions; defined in clause 4.2.8
8. TCP Round Trip Time (Client side); defined in clause 4.4.1
9. TCP Round Trip Time (Server side); defined in clause 4.4.2
10. TCP Retransmission Ratio; defined in clause 4.4.3
11. HTTP Service Setup Time; defined in clause 6.8.4 of [2]
12. HTTP IP-Service Setup Time [s]; see clause 6.8.4 of [2]
13. HTTP Setup Time [s]; s
...