ETSI TR 126 944 V16.1.0 (2021-01)

Universal Mobile Telecommunications System (UMTS); LTE; 5G; End-to-end multimedia services performance metrics (3GPP TR 26.944 version 16.1.0 Release 16)

RTR/TSGS-0426944vg10

General Information

• Status: Not Published
• Technical Committee: 3GPP SA 4 - Codec
• Current Stage: 12 - Completion
• Completion Date: 27-Jan-2021

Standards Content (Sample)

TECHNICAL REPORT
Universal Mobile Telecommunications System (UMTS);
LTE;
5G;
End-to-end multimedia services performance metrics
(3GPP TR 26.944 version 16.1.0 Release 16)

3GPP TR 26.944 version 16.1.0 Release 16 1 ETSI TR 126 944 V16.1.0 (2021-01)

Reference
RTR/TSGS-0426944vg10
Keywords
5G,LTE,UMTS
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2021.
All rights reserved.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.

3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners. ®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 2 ETSI TR 126 944 V16.1.0 (2021-01)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Legal Notice
This Technical Report (TR) has been produced by ETSI 3rd Generation Partnership Project (3GPP).
The present document may refer to technical specifications or reports using their 3GPP identities. These shall be
interpreted as being references to the corresponding ETSI deliverables.
The cross reference between 3GPP and ETSI identities can be found under http://webapp.etsi.org/key/queryform.asp.
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 3 ETSI TR 126 944 V16.1.0 (2021-01)
Contents
Intellectual Property Rights . 2
Legal Notice . 2
Modal verbs terminology . 2
Foreword . 6
Introduction . 6
1 Scope . 8
2 References . 8
3 Abbreviations . 8
4 General description and approach . 9
4.1 Definitions . 9
4.2 QoE parameters . 10
4.3 ESQoS parameters . 11
4.4 SQoS parameters . 11
5 QoE parameters . 13
5.1 General . 13
5.2 Service non-access . 13
5.3 Service failure . 13
5.4 Service setting-up time . 13
5.5 Re-buffering . 13
5.6 Image corruption . 13
5.7 Edge noise . 13
5.8 Blurriness . 14
5.9 Colour reproduction accuracy . 14
5.10 Blockiness . 14
5.11 Incontinuous image with block . 14
5.12 Freeze image . 14
5.13 Audio quality . 14
5.14 Audio/Video synchronization error . 14
6 ESQoS parameters . 14
6.1 General . 14
6.2 Service non-availability . 15
6.2.1 Abstract definition . 15
6.2.2 Equation . 15
6.2.3 Trigger point . 15
6.3 Service failure ratio . 15
6.3.1 Abstract definition . 15
6.3.2 Equation . 15
6.3.3 Trigger point . 16
6.4 Initial connection time . 16
6.4.1 Abstract definition . 16
6.4.2 Equation . 16
6.4.3 Trigger Point . 16
6.5 Initial buffering time . 16
6.5.1 Abstract definition . 16
6.5.2 Equation . 16
6.5.3 Trigger Point . 17
6.6 Re-buffering frequency . 17
6.6.1 Abstract definition . 17
6.6.2 Equation . 17
6.6.3 Trigger point . 17
6.7 Re-buffering time . 17
6.7.1 Abstract definition . 17
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 4 ETSI TR 126 944 V16.1.0 (2021-01)
6.7.2 Equation . 17
6.7.3 Trigger point . 17
6.8 Audio quality . 18
6.8.1 Abstract definition . 18
6.8.2 Equation . 18
6.8.3 Trigger point . 18
6.9 Intra-frame video quality . 18
6.9.1 Abstract definition . 18
6.9.2 Equation . 18
6.9.3 Trigger point . 18
6.10 Inter-frame video quality . 19
6.10.1 Abstract definition . 19
6.10.2 Equation . 19
6.10.3 Trigger point . 19
6.11 Audio/Video de-synchronization error . 19
6.11.1 Abstract definition . 19
6.11.2 Equation . 19
6.11.3 Trigger point . 19
7 SQoS parameters . 20
7.1 SQoS parameters of each layer . 20
7.1.1 Application . 20
7.1.1.1 Frame rate . 20
7.1.1.2 Frame loss ratio . 20
7.1.1.3 Frame error ratio . 20
7.1.1.4 Transaction rate . 20
7.1.1.5 Transaction delay . 20
7.1.2 Transport . 20
7.1.2.1 Bandwidth . 20
7.1.2.2 Transfer delay . 20
7.1.2.3 Jitter. 20
7.1.2.4 Packet loss ratio . 20
7.1.2.5 Packet error ratio . 21
7.1.3 Sub-transport. 21
7.1.3.1 Frame loss ratio . 21
7.1.3.2 Residual frame error ratio . 21
7.1.3.3 Frame discard ratio . 21
7.1.3.4 Residual bit ratio . 21
7.1.3.5 Access delay . 21
7.1.3.6 Block ratio . 21
7.1.3.7 Soft handoff rate . 21
7.2 SQoS parameters of each class . 21
7.3 SQoS parameters of each service . 22
8 Mapping between ESQoS and SQoS . 23
8.1 General . 23
8.2 PSS . 25
8.3 MBMS . 26
8.4 Video telephony . 27
8.5 IMS . 27
Annex A: PSS Performance Analysis with Theoretical Models . 31
A.1 Introduction . 31
A.2 Scope . 31
A.3 References . 31
A.4 General models . 32
A.4.1 Radio access network model . 32
A.4.1.1 Model description . 32
A.4.1.2 The block ratio of new calls and the failure ratio of handoff calls . 33
A.4.2 Transport network model . 34
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 5 ETSI TR 126 944 V16.1.0 (2021-01)
A.4.2.1 Model description . 34
A.4.2.2 The mean transfer delay and the transfer time . 35
A.4.3 Node model . 35
A.4.3.1 Simple node model . 35
A.4.3.1.1 Model description . 35
A.4.3.1.2 The transaction delay and the data loss ratio . 36
A.4.3.2 Complex node model . 36
A.4.3.2.1 Model description . 36
A.4.3.2.2 The transaction delay . 38
A.5 Modelling the relationship between ESQoS and SQoS . 39
A.5.1 Service Non-Availability . 39
A.5.1.1 Relative SQoS and other parameters . 39
A.5.1.2 Model description . 40
A.5.2 Service Failure Ratio . 42
A.5.2.1 Relative SQoS and other parameters . 42
A.5.2.2 Model description . 43
A.5.3 Initial Connection Time . 45
A.5.3.1 Relative SQoS and other parameters . 45
A.5.3.2 Model description . 46
A.5.4 Initial Buffering Time . 48
A.5.4.1 Relative SQoS and other parameters . 48
A.5.4.2 Model description . 49
A.5.5 Re-Buffering Frequency . 51
A.5.5.1 Relative SQoS and other parameters . 51
A.5.5.2 Model description . 51
A.5.6 Re-Buffering Time . 52
A.5.6.1 Relative SQoS and other parameters . 52
A.5.6.2 Model description . 52
Annex B: Analysis of QoE . 53
B.1 Introduction . 53
B.2 Process . 53
B.2.1 Collection of QoE . 53
B.2.2 Weightiness of Selected QoE . 53
B.3 Results . 53
B.3.1 Collection of QoE . 53
B.3.2 Comparison of Weightiness among Selected QoE Parameters . 54
Annex C: Comparison with TS 102 250 of ETSI STQ Mobile . 55
C.1 Introduction . 55
C.2 References . 55
C.3 Comparison with 3GPP E2EMSPM and ETSI STQ Mobile . 56
C.3.1 Research Scope . 56
C.3.1.1 Framework . 56
C.3.1.2 Quality of experience . 56
C.3.1.3 End-to-End Service Quality of Service . 57
C.3.1.4 System Quality of Service . 57
C.3.1.5 Mapping of End-to-End Service QoS and System QoS . 58
C.3.2 Research Method . 58
C.3.2.1 Statistics Model Analysis . 58
C.3.2.2 Theory Model Analysis of Queue . 59
C.3.2.3 Possible Further Work . 59
C.4 Brief Introduction of the Work in ETSI STQ . 59
Annex D: Change history . 60
History . 61
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 6 ETSI TR 126 944 V16.1.0 (2021-01)
Foreword
This Technical Report has been produced by the 3GPP.
The contents of the present document are subject to continuing work within the TSG and may change following formal
TSG approval. Should the TSG modify the contents of the present document, it will be re-released by the TSG with an
identifying change of release date and an increase in version number as follows:
Version 3.y.z
where:
x the first digit:
1 presented to TSG for information;
2 presented to TSG for approval;
3 Indicates TSG approved document under change control.
y the second digit is incremented for all changes of substance, i.e. technical enhancements, corrections,
updates, etc.
z the third digit is incremented when editorial only changes have been incorporated in the specification;
Introduction
The Quality of Experience (QoE), End-to-end Service Quality of Service (ESQoS) and System Quality of Service
(SQoS) are important factors when introducing services to customers. When the next releases of 3G are launched to the
mass market, several new mobile telecommunication multimedia services will be introduced to the general public. It is
essential that a high quality if service is experienced by the user of these new 3G services in order to promote the idea
of 3G as a global all-purpose communication tool for millions of people with widespread availability of terminal
equipment.
The possibility of using multimedia services via 3G in a practical and reliable manner is extremely important in the near
future. For these new services, it is certain that a much larger amount of traffic is generated between mobile terminals
and services, i.e. traffic within 3G networks and between 3G networks and other networks will be higher than has been
the case to date. This gives more importance to service quality requirements.
End-to-end multimedia service performance metrics are proposed to make it possible for operators, device provider and
service providers to more conveniently evaluate their service quality as perceived by end-users. This includes:
- Definition of the performance characteristics that have most relevance to end users (the 'Quality of Experience'
or QoE).
- Definition of the mapping between QoE and end-to-end service measured characteristics (the ESQoS), and
mapping between ESQoS and service specific characteristics (the SQoS).
In the present document a top-down approach is used to illustrate the framework of all metrics.
The present document gives metrics of the end-to-end multimedia service performance on 3G networks that support
PSS, PSC, video telephony, MBMS and IMS services, etc.
QoE parameters describe the end-to-end quality as experienced by the end users. These are difficult to measure and
quantify.
SQoS parameters are metrics that are close related to the network status, and defined from the viewpoint of the service
provider rather than the service user. SQoS parameters can be viewed as the inherent attributes of the networks, which
are important in guaranteeing QoE requirements of the users.
ESQoS parameters describe the QoS of the end-to-end service. They are obtained directly from the QoE parameters by
mapping them into parameters more relevant to operators, service providers and service providers.
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 7 ETSI TR 126 944 V16.1.0 (2021-01)
In Annex A, PSS is taken as an example; several general models looking at different part of network are given. A
mapping between QoE, ESQoS and SQoS is given using a mathematical approach, which defines a dimensioned
relationship between user experiences and lower tiers of network performance.
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 8 ETSI TR 126 944 V16.1.0 (2021-01)
1 Scope
The present document describes and defines performance metrics for popular multimedia services in 3G networks,
including packed-switched streaming service (PSS), multimedia broadcast multicast service (MBMS), video telephony
(VT), and IP multimedia subsystem service (IMS). The present document has a top-down approach, which starts with
the Quality of Experience (QoE) parameters and metrics, and then provides End-to-end Service QoS (ESQoS) and
System Quality of Service (SQoS) parameters and metrics and mapping between these different layers.
2 References
The following documents contain provisions which, through reference in this text, constitute provisions of the present
document.
- References are either specific (identified by date of publication, edition number, version number, etc.) or
non-specific.
- For a specific reference, subsequent revisions do not apply.
- For a non-specific reference, the latest version applies. In the case of a reference to a 3GPP document (including
a GSM document), a non-specific reference implicitly refers to the latest version of that document in the same
Release as the present document.
[1] 3GPP TS 23.107:"Technical Specification Group Services and System Aspects; Quality of Service
(QoS) concept and architecture".
[2] 3GPP TS 23.207:"Technical Specification Group Services and System Aspects; End-to-end
Quality of Service (QoS) concept and architecture".
[3] ETSI TS 102 250-2 V1.4.1 (2006-03): "Speech Processing, Transmission and Quality Aspects
(STQ);QoS aspects for popular services in GSM and 3G networks; Part 2: Definition of Quality of
Service parameters and their computation".
[4] IETF RFC 2679: "A One-way Delay Metric for IPPM".
[5] IETF RFC 3133: "Terminology for Frame Relay Benchmarking".
[6] ITU-R Recommendation BS.1387-1: "Method for objective measurements of perceived audio
quality".
[7] 3GPP TS 26.234: "Technical Specification Group Services and System Aspects; Transparent
end-to-end Packet-switched Streaming Service (PSS); Protocols and codecs".
[8] 3GPP TS 26.346: "Technical Specification Group Services and System Aspects; Multimedia
Broadcast/Multicast Service (MBMS); Protocols and codecs".
[9] ITU-T Recommendation P.10/G.100: "Vocabulary for performance and quality of service".
3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
AS Application Server
BGCF Breakout Gateway Control Function
BM-SC Broadcast-Multicast - Service Centre
CBC Cell Broadcast Centre
CSCF Call Session Control Function
DHCP Dynamic Host Configuration Protocol
DNS Domain Name System
ESQoS End-to-end Service QoS
GGSN Gateway GPRS Support Node
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 9 ETSI TR 126 944 V16.1.0 (2021-01)
HSS Home Subscriber Server
I-CSCF Interrogating-CSCF
IMS IP Multimedia Subsystem
IP-CAN IP-Connectivity Access Network
LAV Least Acceptable Value
MBMS Multimedia Broadcast/Multicast Service
MGCF Media Gateway Control Function
MGW Media GateWay
P-CSCF Proxy-CSCF
PSC Packet-Switched Conversational service
PSS Packed-Switched Streaming service
QoE Quality of Experience
QoS Quality of Service
SQoS System Quality of Service
UE User Equipment
VLR Visitor Location Register
VT Video Telephony
4 General description and approach
4.1 Definitions
Quality of Experience (QoE)
The overall acceptability of an application or service, as perceived subjectively by the end-user (see notes 1 and 2).
NOTE 1: Quality of Experience includes the complete end-to-end system effects (client, terminal, network, services
infrastructure, etc).
NOTE 2: Overall acceptability may be influenced by user expectations and context.
For the purpose of the present document, QoE is limited to those parameters that are reasonably expected to be under
the control of the Service Provider, i.e. it excludes aspects of specific terminal equipment implementation. In the
document, the exact E2E measurement point will be defined for each service.
QoE indicates performance metrics as expressed from the end service user's point of view. They can be required or
reported by the common users, and may be stated irrespective of their measurability.
SA4 has defined Quality of Experience (QoE) metrics and their transport for PSS and MBMS in [7] and [8],
respectively. The PSS and MBMS QoE metrics features are optional for both PSS and MBMS streaming server and
clients, and shall not disturb the PSS and MBMS service. A PSS or MBMS client supporting the feature shall perform
the quality measurements in accordance to the measurement definitions, aggregate them into client QoE metrics and
report the metrics to the PSS or MBMS server using described procedures in [7] and [8] for PSS and MBMS,
respectively. The way the QoE metrics are processed and made available is out of the scope of the present document.
In general, a Service Provider will set service requirements in line with the end-user's expected QoE, which needs to be
translated into parameters or metrics that the service provider can control or measure. Thus, it is necessary to map QoE
metrics to measurable End-to-end Service QoS and System QoS parameters, which provide the means for the service
providers to guarantee the service quality.
End-to-end Service QoS (ESQoS)
ESQoS is generally used to specify the performance of services from the perspective of operators and service providers.
As the number of new services proliferates and becomes more complex, it is important for an operator to measure a
network's ESQoS accurately and continuously improve it to achieve customer loyalty and maintain competitive edge.
ESQoS is measurable. It can be quantified exactly by several digital parameters, unlike QoE.
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 10 ETSI TR 126 944 V16.1.0 (2021-01)
System Quality of Service (SQoS)
The concept of SQoS is a subset of conventional QoS, which is defined in ITU-T Recommendation E.800 as the
collective effect of service performances which together determines the satisfaction of a user of a service. It is
characterized by the combined aspects of performance factors applicable to all services, such as:
- service support performance;
- service operability performance;
- service accessibility performance;
- service retainability performance;
- service integrity performance;
- service security performance;
Compared with ESQoS, SQoS denotes the point-to-point QoS, which is specifically related to the units and links of
network systems, rather than the whole service or a network. SQoS can be viewed as the QoS consideration from the
viewpoint of network operators.
4.2 QoE parameters
From the user perspective, the QoE cannot be defined only from technical measurements. It should be defined by user
investigation using the following methods:
1. Users provide an opinion of QoE only with their feeling and experience. The QoE metric is subjective.
2. With some classic indicators of QoE, users can make a decision to choose some of them as the necessary
indicators of QoE.
Service performance is characterized by the following:
Because QoE parameters and metrics directly relate to the definition of the service itself, it is possible to map the
components of the service onto measurable ESQoS and SQoS parameters or metrics, as these relate directly to the
performance of the components of the service.
This concept can be illustrated by considering the example of a video service, where QoE includes items such as:
- service setup delay;
- re-buffering duration;
- end to end delay;
- corruption duration;
- mean time between corruption;
- content quality(e.g. digital TV-like quality, analog TV-like quality, DSL-like video conference quality ISDN-
like video conference quality, etc.);
- audio/video synchronization (or 'lip sync');
- service availability.
Common user requirements for different services are therefore collected and classified as our QoE, disregarding the
measurability.
Users have different requirements for different services. So the QoE for service analysis varies with the types of
services, methods of delivery and points of monitoring.
Figure 1 shows the model.
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 11 ETSI TR 126 944 V16.1.0 (2021-01)

Figure 1: QoE
4.3 ESQoS parameters
As discussed, the user's definition of QoE does not consider measurability and so ESQoS parameters need to be
introduced. These express QoS from the viewpoint of operators and service providers. ESQoS parameters are from end-
to-end perspective, i.e. across the full set of components which together provide a service, and, most importantly can be
measured and quantified.
ESQoS parameters depend on the type of service, and so there will be a set corresponding to each service.
Figure 2 shows the model including both the QoE and ESQoS aspects.

Figure 2: QoE and ESQoS parameters
4.4 SQoS parameters
The quality of a service perceived by the user perceives is influenced by the network as well as by the terminal
equipment. As the intelligence and complexity of an application used in 3G increases, application performance becomes
an important factor in determining the quality perceived by the user. Within the present document, the influence of the
terminal equipment and application have not been included.
Services are applications running on the common network components. SQoS parameters are specific metrics that
provide measurements which reflect network component status, and are defined from the viewpoint of the service
provider rather than the service user.
SQoS parameters can be classified to five classes as shown in figure 3.
ETSI
3GPP TR 26.944 version 16.1.0 Release 16 12 ETSI TR 126 944 V16.1.0 (2021-01)
NOTE: The network entities in the core network (CN) can be considered as servers.

Figure 3: Five classes' scopes
Each class also has three layers which reflect the protocol layers in the network.
- The first s the Application layer.
- The second is the Transport layer.
- The third consists of Network, Data link and Physical layers.
Figure 4 presents the overall model.
Users
QoE
PSS MBMS VT IMS
End-to-end Service QoS
PSS MBMS VT IMS
Application
Transport System QoS
Vertical
Under
transport
Wireless
Wired transport Transaction
Client Radio access transport
network Node
network
Horizontal
Figure 4: QoE, ESQoS and SQoS parameters
ETSI
...