ETSI TR 103 488 V1.1.1 (2019-01)

Speech and Multimedia Transmission Quality (STQ); Guidelines on OTT Video Streaming; Service Quality Evaluation Procedures

DTR/STQ-00215m

General Information

• Status: Published
• Publication Date: 23-Jan-2019
• Technical Committee: STQ MOBILE - STQ Mobile
• Current Stage: 12 - Completion
• Due Date: 08-Feb-2019
• Completion Date: 24-Jan-2019

Standards Content (Sample)

TECHNICAL REPORT
Speech and Multimedia Transmission Quality (STQ);
Guidelines on OTT Video Streaming;
Service Quality Evaluation Procedures

2 ETSI TR 103 488 V1.1.1 (2019-01)

Reference
DTR/STQ-00215m
Keywords
3G, data, GSM, network, QoE, QoS, service,
video
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2019.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners. ®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI
3 ETSI TR 103 488 V1.1.1 (2019-01)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
Executive summary . 4
Introduction . 4
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definition of terms, symbols and abbreviations . 7
3.1 Terms . 7
3.2 Symbols . 7
3.3 Abbreviations . 7
4 Passive, non-intrusive network monitoring of SSL/QUIC OTT video services bitstreams . 8
5 Categories of OTT video streaming session QoS parameters . 8
5.1 Introduction . 8
5.2 Triggers of main QoS parameters . 10
5.2.1 Session . 10
5.2.2 Video Access Time and Pre-playout buffering time . 10
5.3 Parameters relevance for an effective Root Cause Analysis . 10
5.3.1 Transport parameters relevance . 10
5.3.2 QoE-Video/Audio parameters relevance . 11
5.3.3 Streaming session parameters relevance . 11
5.3.4 Service centric parameters relevance . 11
6 Thresholds for OTT video streaming quality . 11
7 Figure of Merit of the OTT video streaming service . 12
7.1 Introduction . 12
7.2 Calculation of Figure of Merit . 13
7.3 Quality thresholds used for FoM calculation . 14
7.4 Weightings . 14
7.5 Video streaming sessions' quality classification . 14
8 Reporting . 15
Annex A: Example of FoM calculation . 16
Annex B: Bibliography . 18
History . 19

ETSI
4 ETSI TR 103 488 V1.1.1 (2019-01)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Speech and multimedia Transmission
Quality (STQ).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Executive summary
The fast increase of the variety, technology complexity, and dynamic changes of the OTT video streaming services'
delivery, as well as their consumptions by the users on a wide range of smart devices, suggests a consistent and robust
testing approach that can allow an application transparent and meaningful evaluation of these services' quality
management and control. The present document offers guidance for such a testing approach, while aligning the content
with all other ETSI WG STQ Mobile documents as well as work related to the topic ongoing in other organizations.
Introduction
Several published STQM TSs and TRs cover the area of the video streaming end to end QoS/QoE evaluation. ETSI
TS 102 250-2 [i.1] covers video streaming session performance evaluation in the case of real time streaming (e.g. RTP,
RTSP, RTCP). ETSI TR 101 578 [i.2] complements ETSI TS 102 250-2 [i.1] with QoS/QoE for the specific case of
TCP based streaming (YouTube™ application), which is the de-facto delivery technique for the OTT video streaming
applications. Last, but not least, ETSI TR 102 493 [i.3] provides guidance on available QoE algorithms that can be used
in various video streaming testing scenarios.
ETSI
5 ETSI TR 103 488 V1.1.1 (2019-01)
However, the OTT video streaming services' arena has seen video technology evolution (e.g. H.265 codec, 1440
resolution, new protocols such as QUIC), as well as dynamic technology changes (e.g. encryption schemes, encoding
schemes of various profiles, adaptations schemes at the server and/or client side within the context of various throttling
techniques and policies). Last, but not least, the variety of devices, namely operation system based video clients, do see
the same dynamic change. Therefore, guidance for a transparent and flexible testing approach is required by today's
OTT video streaming sessions' quality evaluation. And, consequently, a set of the most meaningful QoE centric QoS
parameters is also necessary, which can complement MOS estimators (QoE algorithms/models; Recommendations
ITU-T J series [i.4] to [i.9], P series [i.10] and [i.11]) whenever these are available, but also go beyond a single quality
score.
ETSI
6 ETSI TR 103 488 V1.1.1 (2019-01)
1 Scope
The present document's scope is to provide guidance on OTT video streaming testing approach with a set of minimum
desired and most meaningful QoE centric QoS parameters along with recommendations to create a figure of merit
quantifying the OTT video streaming session quality, where possible. In addition, the set of introduced QoE centric
QoS parameters aim to help with the identification of the possible roots of video quality degradation. The present
document also offers means to understand aspects related with network and services optimization and troubleshooting,
such as the trade-off between bandwidth usage or controlled throttling and end-to-end video quality.
The scope of the present document complements ETSI TS 102 250-2 [i.1] and ETSI TR 101 578 [i.2] while not being as
exhaustive, but rather focused on QoE centric characterization and an end-to-end view on the video streaming session
as a whole. Furthermore, the present document takes into consideration QoE centric evaluation by means of passive,
non-intrusive network monitoring of SSL/QUIC OTT Video Services bitstreams. In addition, the present document
aims to complement the scope of ETSI TR 102 493 [i.3] with respect to QoE models for video streaming integrity as
perceived by users.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
[i.2] ETSI TR 101 578: "Speech and multimedia Transmission Quality (STQ); QoS aspects of TCP-
based video services like YouTube™".
[i.3] ETSI TR 102 493: "Speech and multimedia Transmission Quality (STQ); Guidelines for the use of
Video Quality Algorithms for Mobile Applications".
[i.4] Recommendation ITU-T P.1201: "Parametric non-intrusive assessment of audiovisual media
streaming quality".
[i.5] Recommendation ITU-T P.1202: "Parametric non-intrusive bitstream assessment of video media
streaming quality".
[i.6] Recommendation ITU-T P.1203: "Parametric bitstream-based quality assessment of progressive
download and adaptive audiovisual streaming services over reliable transport".
[i.7] Recommendation ITU-T P.343: "Hybrid perceptual bitstream models for objective video quality
measurements".
[i.8] Recommendation ITU-T P.341: "Objective perceptual multimedia video quality measurement of
HDTV for digital cable television in the presence of a full reference".
ETSI
7 ETSI TR 103 488 V1.1.1 (2019-01)
[i.9] Recommendation ITU-T P.342: "Objective multimedia video quality measurement of HDTV for
digital cable television in the presence of a reduced reference signal".
[i.10] Recommendation ITU-T J.247: "Objective perceptual multimedia video quality measurement in
the presence of a full reference".
[i.11] Recommendation ITU-T J.246: "Perceptual visual quality measurement techniques for multimedia
services over digital cable television networks in the presence of a reduced bandwidth reference".
[i.12] Recommendation ITU-T P.1401: "Methods, metrics and procedures for statistical evaluation,
qualification and comparison of objective quality prediction models".
[i.13] Larry Stephens: "Schaum's Outline of Statistics" series, McGraw-Hill Trade, January 1989.
3 Definition of terms, symbols and abbreviations
3.1 Terms
Void.
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Programming Interface
CDN Content Delivery Network
DASH Dynamic Adaptive Streaming over HTTP
DL DownLoad
FoM Figure of Merit
IP Internet Protocol
MOS Mean Opinion Score
OS Operating System
OTT Over The Top
QoE Quality of Experience
QoS Quality of Services
QUIC Quick UDP Internet Connections
RCA Root Cause Analysis
RTCP Real Time Control Protocol
RTP Real Time Protocol
RTSP Real Time Streaming Protocol
SSL Secure Sockets Layer
TCP Transport Control Protocol
TH THreshold
UDP User Datagram Protocol
UE User Equipment
WG Working Group
ETSI
8 ETSI TR 103 488 V1.1.1 (2019-01)
4 Passive, non-intrusive network monitoring of
SSL/QUIC OTT video services bitstreams
OTT Video payloads transported by SSL and QUIC, together with metadata encryption, reduce the applicability of QoE
algorithms/models referred by ETSI TR 102 493 [i.3] and Recommendation ITU-T J series, P series [i.4], [i.5], [i.6],
[i.7], [i.8], [i.9], [i.10] and [i.11] for non-intrusive network monitoring at mid-point solutions and prevent them to
provide the MOS identified by such standards.
The approach described in the present document is based on the figure of merit and on identification of the possible
causes of video quality degradation, complements the one depicted by ETSI TR 102 493 [i.3] for non-intrusive network
monitoring solutions and provides a reference for them when payloads are transported by SSL and QUIC.
Note that such approach can be also extended to all the passive monitoring solutions, including any possible ones on the
User Equipment, when only transported payloads by SSL and QUIC are available.
5 Categories of OTT video streaming session QoS
parameters
5.1 Introduction
The whole end-to-end OTT video streaming session quality is determined and impacted by four categories of QoS
parameters:
• Transport/delivery QoS parameters.
• Audio/video integrity QoS parameters.
• Streaming session QoS parameters.
• Service centric QoS parameters.
Each category can cover a set of QoS parameters which can be either measured and/or calculated directly, or can be
estimated based on inferences (that are out of the scope of the present document) made on client's events, service's
states and transport packet analysis. The latter is needed due to various types and levels of encryptions. At the time of
publication of the present document, there are scenarios within which even these inferences techniques are challenged
by fully encrypted streams, such as QUIC protocol, which leaves no metadata available. Work is going on in 3GPP,
IETF and DASH Industry Forum in order to offer an open API interface which would allow access to minimum
required metadata.
Table 1 provides a list of minimum required and most meaningful QoS parameters which can be used for end-to-end
characterization (e.g. CDNs Id, CDN protocol, subscriber IP address, session id, video provider) and quantification
(e.g. video-audio QoS parameters, streaming QoS parameters) of the OTT video streaming session' s quality.Ultimately,
these QoS parameters can be further used to troubleshoot and identify more likely root causes of possible quality
problems.
The columns in table 1 refer to:
• Category: Transport; audio/video; streaming session; Service centric.
• Name: It provides the parameter name.
• Type: It refers to QoS or Auxiliary (Aux) aspects.
ETSI
9 ETSI TR 103 488 V1.1.1 (2019-01)
Table 1: QoS parameters
Category Name Description Type
Transport Number of CDN Media Number of participating CDN Media Server inside a Aux
Servers session.
CDN Media ServerX Name of the used CDN Media Server X. Aux
Name X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Media ServerX IP IP Address of the used CDN Media Server X. Aux
address X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Media ServerX Bytes downloaded from CDN Media server X Aux
downloaded bytes X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Media ServerX Average time between the last packet of the player QoS
Average Time to first request till the first packet of the related CDN Media
packet (CDN delay) server response for all the player requests with a
response.
X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Media ServerX Rate of Player Requests with no answer towards CDN QoS
Failure rate (%) Media ServerX.
X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Downlink It is the overall Downlink Application Throughput related QoS
Application throughput to media content downloaded from CDN Media Servers
(Kbps) (Kbps).
Video/Audio Avg. video buffer (s) Average Player video buffer size during play time in Aux
second.
Avg. audio buffer (s) Average Player Audio buffer size during play time in Aux
second.
Avg. video bit rate Average video bit rate during play time (Kbps). QoS
(Kbps)
Avg. video bit rate Average video bit rate range during play time (low, fair, QoS
Range excellent). See clause 6 for Range definition.
Avg. audio bit rate Avg. audio bit rate during play time (low, fair, excellent). QoS
Range See clause 6 for range definition.
Low Video bit rate % Play time % at Low Video Bit Rate range. See clause 6 QoS
for range definition.
Low Audio bit rate % Play time % at Low Audio Bit Rate range. See clause 6 QoS
for range definition.
Good Video bit rate % Play time % at Good Video Bit Rate range. See clause 6 QoS
for range definition.
Good Audio bit rate % Play time % at Good Audio Bit Rate range. See clause 6 QoS
for range definition.
Excellent Video bit rate Play time % at Excellent Video bit rate range. See QoS
% clause 6 for range definition.
Excellent Audio bit rate Play time % at Excellent Audio bit rate range. See QoS
% clause 6 for range definition.
Video bit rate range Number of Video bit rate range switches. See QoS
switches clause 6 for range definition.
Audio bit rate range Number of Audio bit rate range switches. See QoS
switches clause 6 for range definition.
Positive Video bit rate Number of switches to higher video bit rate range. See QoS
range switches Clause 6 for range definition.
Positive Audio bit rate Number of switches to higher audio bit rate range. See QoS
range switches clause 6 for range definition.
Negative Video bit rate Number of switches to lower video bit rate range. See QoS
range switches clause 6 for range definition.
Negative Audio bit rate Number of switches to lower audio bit rate range. See QoS
range switches clause 6 for range definition.
Video DL Mbytes Video downloaded Mbytes. Aux
Audio DL Mbytes Audio downloaded Mbytes. Aux
Video Quality It refers to ETSI TR 101 578 [i.2] Video Quality QoS
parameter.
ETSI
10 ETSI TR 103 488 V1.1.1 (2019-01)
Category Name Description Type
streaming Video Access Time (s) It refers to ETSI TR 101 578 [i.2] Video Access Time (s) QoS
session parameter, see also clause 5.1.
Pre-Playout Buffering It refers to ETSI TR 101
...