ETSI EG 202 009-2 V1.3.1 (2014-12)

User Group; Quality of telecom services; Part 2: User related indicators on a service specific basis

REG/USER-00042-2

General Information

• Status: Published
• Publication Date: 01-Dec-2014
• Technical Committee: USER - User Group
• Current Stage: 12 - Completion
• Due Date: 12-Dec-2014
• Completion Date: 02-Dec-2014

Standards Content (Sample)

ETSI EG 202 009-2 V1.3.1 (2014-12)






ETSI GUIDE
User Group;
Quality of telecom services;
Part 2: User related indicators on a service specific basis

---------------------- Page: 1 ----------------------
2 ETSI EG 202 009-2 V1.3.1 (2014-12)



Reference
REG/USER-00042-2
Keywords
QoS, quality, service, SLA, user
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2014.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI EG 202 009-2 V1.3.1 (2014-12)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 7
2 References . 7
2.1 Normative references . 8
2.2 Informative references . 8
3 Definitions and abbreviations . 10
3.1 Definitions . 10
3.2 Abbreviations . 10
4 QoS methodology implementation . 11
4.1 General principles for the metric and indicator definition . 11
4.2 QoS requirements vs service element . 12
4.2.1 Flexibility . 12
4.2.2 Usability . 13
4.2.3 Security . 13
5 QoS metrics and indicators for the service behaviour in the service utilization . 13
5.1 Audio broadcast - Audiostreaming . 14
5.2 Directory enquiry services . 15
5.3 E-mail . 16
5.4 Fax . 18
5.5 Internet services . 19
5.6 Multimedia Message Service (MMS). 23
5.7 Operator services . 25
5.8 Short Message Service (SMS) . 26
5.9 Telephony . 28
5.10 Video broadcast - Video streaming . 31
5.11 Voice mail . 33
5.12 Voice messaging . 34
5.13 For further study . 35
6 QoS metrics and indicators for all steps of the customer relationship course other than utilization . 36
6.1 Sales . 36
6.1.1 Preliminary information . 36
6.1.1.1 Preliminary information needed by the customer for a telephony contract (fixed or mobile) . 38
6.1.1.2 Preliminary information needed by the customer for an ISP contract . 38
6.1.1.3 Preliminary information needed by the customer for a multi-services contract (fixed or mobile) . 39
6.1.2 Establishment of the contract (Terms and conditions) . 39
6.2 Service provisioning . 41
6.3 Service alteration and technical upgrade . 44
6.3.1 Service alteration . 44
6.3.2 Technical upgrade . 46
6.4 Service support . 47
6.4.1 Documentation . 48
6.4.2 Technical support . 49
6.4.3 Commercial support . 50
6.4.4 Complaint management . 52
6.5 Repair services . 54
6.6 Metering/charging/billing . 56
6.7 Network/service management by the customer . 59
6.8 Cessation . 61
7 Specific service elements . 62
ETSI

---------------------- Page: 3 ----------------------
4 ETSI EG 202 009-2 V1.3.1 (2014-12)
7.1 Security as a service element . 62
7.1.1 Identification service element . 63
7.1.2 Authentication service element . 64
7.1.3 Authorization service element . 64
7.1.4 Encryption service element . 65
7.1.5 Non repudiation service element . 66
7.1.6 Time stamping service element . 67
7.1.7 Digital signature service element . 68
7.1.8 Certificate management service element . 68
7.2 Flexibility as a service element . 69
7.2.1 (Re)configuration service element . 70
7.2.2 (Re)provisionning service element . 70
8 Specific aspects of the general public users' criteria . 71
9 Conclusion . 72
Annex A: Bibliography . 73
History . 75

ETSI

---------------------- Page: 4 ----------------------
5 ETSI EG 202 009-2 V1.3.1 (2014-12)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI User Group (USER).
It includes, among other contributions, excerpts of the final report of Bannock Consulting's project for the European
Commission's DG Information Society.
The present document is part 2 of a multi-part deliverable covering the quality of telecom services, as identified below:
Part 1: "Methodology for identification of indicators relevant to the Users";
Part 2: "User related indicators on a service specific basis";
Part 3: "Template for Service Level Agreements (SLA)".
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "may not", "need", "need not", "will",
"will not", "can" and "cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms
for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
Quality of Service can be evaluated from different perspectives and therefore using different measurement methods:
a) a first level of QoS is related to the reliability of the equipment and can be measured accurately via technical
means;
b) a second level is related to the service provisioning and is closely linked to the kind of use of the service.
Therefore appropriate criteria have to be defined according to this kind of use between the customer and the
supplier; the service delivery QoS depends on the network as well as on the server behaviours;
c) the last one is intended to measure the subjective satisfaction of the customer and there is often no other means
than a survey to get it (MOS value for media quality and OR for other services).
In the two first categories, technical means can be used to perform the measurements and in such cases, standards are
often useful to achieve a common approach; such standards are given as references where appropriate. They include a
precise definition of the relating metric and indicators and how to measure them. Unfortunately, they are not always
providing enough indications on the size and how to select the samples to be measured. ETSI EG 202 009-1 [i.11] can
help on these aspects but additional study may be needed to reach the right accuracy.
In the last category, the present document aims to give guidance on how to carry out the measurements including the
subjective ones.
ETSI

---------------------- Page: 5 ----------------------
6 ETSI EG 202 009-2 V1.3.1 (2014-12)
Measurements of every interesting indicator all the time might be very expensive and can even jeopardize the network
and service performances. It can be more appropriate to get some of them via a poll on a limited number of users and
for a limited period of time. In addition, a third party may be needed to carry out these measurements to make them
more reliable and avoid any criticism from one of the involved parties.
ETSI

---------------------- Page: 6 ----------------------
7 ETSI EG 202 009-2 V1.3.1 (2014-12)
1 Scope
In the current competitive world, Quality of Service (QoS) is becoming, jointly with cost, a key parameter in selling and
buying telecommunications services. At the same time, technology and liberalization trends are raising new types of
concerns unknown with the Plain Old Telephony Services (POTS) using switched connections provided by a single
monopoly supplier.
Nowadays, there are several standards describing QoS measurements but the question of which indicators are the most
interesting to be monitored from the users point of view and which values they should meet is still open. The present
document proposes a reference model to evaluate the Quality of Service from the users point of view, defining the
following concepts:
a) the appropriate indicators for a QoS assessment from the user point of view, i.e. KQI and SLO (see ETSI
EG 202 009-1 [i.11], clause 4.3);
b) the methods to acquire the indicator values (KQI) needed to assess the quality of service.
The main principles for these definitions are:
• To define the services according to the applications performed by the user and not by the technical solution:
for example, voice over IP is one of the many technical solutions to communicate between subscribers of the
world-wide telephone network; ATM, frame Relay, IP are some of the many technical solutions to ensure a
data transmission service between a terminal and a server or between networks. The quality criteria are the
same, only the Service Level achieved can be different.
• To define the quality criteria with respect of usage and not technique. In speech quality, users are more
interested in intelligibility than in bandwidth, distortion, signal to noise ratio or lost packets. Quality criteria
should be defined as SLO (KQI), then translated into technical criteria (KPI). This means that providers may
use different performance indicators to quantify and monitor the quality, depending on the technology used to
match a particular SLO.
Therefore, the present document does not intend to describe measurement techniques since several ETSI TCs are
dealing with such techniques and have the appropriate technical knowledge to develop standards in this area.
ETSI EG 202 009-1 [i.11] gives guidance in identifying the indicators relevant from the user requirement point of view.
If these metrics and indicators are used in a Service Level Agreements (SLA), it is crucial to define, at least for the most
important ones, the agreed quality targets. If the aim is a comparison of the respective providers' performances, then
quality targets can be provided as guidance for the general public to assess if the QoS of the results achieved is
satisfactory or not. The preferred values and the means to aggregate them are provided in a separate document
ETSI EG 202 934 [i.17].
The present document intends to define user related service specific KQI as far as possible using formal standards while
ETSI EG 202 009-3 [i.12] proposes a template for a SLA dealing with all service aspects, including penalties, escalation
procedures, areas of responsibility, etc. where these metrics and indicators can be used.
The purpose of the present document is to use the methodology described in ETSI EG 202 009-1 [i.11] to define, for
each QoS criterion, the relevant metrics and indicators (KQI) for a choice of services and for each step of the customer
relationship course. Hence each customer can have a comprehensive information on the features of the service he
intends to buy according to the various providers. This will enable him to select the best suited to his needs.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
reference document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
ETSI

---------------------- Page: 7 ----------------------
8 ETSI EG 202 009-2 V1.3.1 (2014-12)
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] Recommendation ITU-T G.1010: "End-user multimedia QoS categories".
[i.2] Recommendation ITU-T P.800: "Methods for subjective determination of transmission quality".
[i.3] Recommendation ITU-T P.800.1: "Mean Opinion Score (MOS) terminology".
[i.4] Recommendation ITU-T P.862: "Perceptual evaluation of speech quality (PESQ), an objective
method for end-to-end speech quality assessment of narrowband telephone networks and speech
codecs".
[i.5] Recommendation ITU-T P.862.1: "Mapping function for transforming P.862 raw result scores to
MOS-LQO".
[i.6] Recommendation ITU-T P.862.2: "Wideband extension to Recommendation P.862 for the
assessment of wideband telephone networks and speech codecs".
[i.7] Recommendation ITU-T P.862.3: "Application guide for objective quality measurement based on
Recommendations P.862, P.862.1 and P.862.2".
[i.8] Recommendation ITU-T P.863: "Perceptual objective listening quality assessment".
[i.9] Recommendation ITU-T T.22: "Standardized test charts for document facsimile transmissions".
[i.10] ETSI EG 201 769: "Speech Processing, Transmission and Quality Aspects (STQ); QoS parameter
definitions and measurements; Parameters for voice telephony service required under the ONP
Voice Telephony Directive 98/10/EC".
[i.11] ETSI EG 202 009-1: "User Group; Quality of telecom services; Part 1: Methodology for
identification of parameters relevant to the Users".
[i.12] ETSI EG 202 009-3: "User Group; Quality of telecom services; Part 3: Template for Service Level
Agreements (SLA)".
[i.13] ETSI EG 202 057-2: "Speech and multimedia Transmission Quality (STQ); User related QoS
parameter definitions and measurements; Part 2: Voice telephony, Group 3 fax, modem data
services and SMS".
[i.14] ETSI EG 202 057-3: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 3: QoS parameters specific to Public Land
Mobile Networks (PLMN)".
[i.15] ETSI EG 202 057-4: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 4: Internet access".
[i.16] ETSI EG 202 843: "User Group; Quality of ICT services; Definitions and methods for assessing
the QoS parameters of the customer relationship stages other than utilization".
[i.17] ETSI EG 202 934: "User Group; The assessment of the overall Quality of Services (QoS) as
perceived by the users; Definition of QoS indexes for all the customer relationship stages".
[i.18] ETSI ES 202 057-1: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 1: General".
ETSI

---------------------- Page: 8 ----------------------
9 ETSI EG 202 009-2 V1.3.1 (2014-12)
[i.19] ETSI ES 202 765-2: "Speech and multimedia Transmission Quality (STQ); QoS and network
performance metrics and measurement methods; Part 2: Transmission Quality Indicator combining
Voice Quality Metrics".
[i.20] ETSI ES 202 765-4: "Speech and multimedia Transmission Quality (STQ); QoS and network
performance metrics and measurement methods; Part 4: Indicators for supervision of Multiplay
services".
[i.21] ETSI TR 102 806: "User Group; Analysis of current End-to-End QoS standardization state".
[i.22] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
[i.23] ETSI TS 102 844: "User Group; Quality of Telecom Services; Conformity assessment;
Requirements for bodies providing QoS assessments and surveys".
[i.24] CEN/CENELEC/ETSI EN 301 549: "Accessibility requirements suitable for public procurement
of ICT products and services in Europe".
[i.25] CEN CWA14357: "CEN Workshop Agreement -Quality of Internet Service - Project Team Final
Report - ICS 35.240.60".
[i.26] IETF RFC 792: "Internet Control Message Protocol".
[i.27] ANSI/ASA S3.5-1997 (R2012): "American National Standard Methods for Calculation of the
Speech Intelligibility Index".
[i.28] Quality of Service Parameters for Internet Service Provision, final report of Bannock Consulting's
project for the European Commission's DG Information Society.
[i.29] ISO/IEC 18028-4:2005: "Information technology -- Security techniques -- IT network security --
Part 4: Securing remote access".
[i.30] ISO/IEC 7498-2: "Information processing systems -- Open Systems Interconnection -- Basic
Reference Model -- Part 2: Security Architecture".
[i.31] ISO/IEC 9797-1: "Information technology -- Security techniques -- Message Authentication Codes
(MACs) -- Part 1: Mechanisms using a block cipher".
[i.32] ISO/IEC 13888-1: "Information technology -- Security techniques -- Non-repudiation -- Part 1:
General".
[i.33] ISO/IEC 15945:2002: "Information technology -- Security techniques -- Specification of TTP
services to support the application of digital signatures".
[i.34] ISO/IEC 11770-3:1999: "Information technology -- Security techniques -- Key management --
Part 3: Mechanisms using asymmetric techniques".
[i.35] ITSEC: "Information Technology Security Evaluation Criteria - Provisional Harmonized Criteria"
June 1991.
ETSI

---------------------- Page: 9 ----------------------
10 ETSI EG 202 009-2 V1.3.1 (2014-12)
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the terms and definitions given in ETSI EG 202 009-1 [i.11] apply.
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ADSL Asymmetric Digital Subscriber Line
ATM Asynchroneous Transfer Mode
CRM Customer Relationship Management
DNS Domain Name System
FTP File Transfer Protocol
GPRS General Packet Radio Service
GSM Global System for Mobile
HTTP Hyper Text Transfer Protocol
ICMP Internet Control Management Protocol
ICT Information and Communication Technologies
IP Internet Protocol
IPTV Television over Internet Protocol
ISDN Integrated Services Digital Network
ISP Internet Service Provider
IT Information Technology
ITSEC Information Technology SECurity
IVR Interactive Voice Response
KPI Key Performance Indicator
KQI Key Quality Indicator
MMS Multimedia Message Service
MMSC Multimedia Messaging Service Centre
MO Mobile Originate
MOS Mean Opinion Score
MT Mobile Terminate
OR Opinion Rating
PABX Private Automatic Branch eXchange
PDD Post Dialling Delay
PI Preliminary Information
PoP Point of Presence
POTS Plain Old Telephony Service
PSTN Public Switched Telephone Network
PTN Private Telecommunications Networks
QoS Quality of Service
RFC Request For Comment
SDS Short Data Service
SLA Service Level Agreement
SLO Service Level Objective
SMS Short Message Service
SP Service Provider
ST-MOS Listen Speech Quality Stability
TV Television
UE User Equipment
UMTS Universal Mobile Telecommunications System
WGR WAP Get Request
ETSI

---------------------- Page: 10 ----------------------
11 ETSI EG 202 009-2 V1.3.1 (2014-12)
4 QoS methodology implementation
To implement the methodology detailed in clause 6 of ETSI EG 202 009-1 [i.11], the process consists first in defining
carefully the intended service, including its conditions of use and service level range, then to check for each step of the
customer relationship course and each QoS criterion what are the best suited metrics and indicators to express the user's
requirements and to monitor these indicators appropriately. When this is done for each cell of the ETSI
EG 202 009-1 [i.11] matrix, there will probably be too much indicators for a convenient handling, therefore only the
most relevant should be taken for publication or to include in a SLA.
Trying to limit drastically the number of indicators may be counter-productive as it gives the provider an incentive to
focus on a particular measure, perhaps at the expense of the general QoS.
...

Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)






ETSI GUIDE
User Group;
Quality of telecom services;
Part 2: User related indicators on a service specific basis

---------------------- Page: 1 ----------------------
2 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)



Reference
REG/USER-00042-2
Keywords
QoS, quality, service, SLA, user
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2014.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 7
2 References . 7
2.1 Normative references . 8
2.2 Informative references . 8
3 Definitions and abbreviations . 10
3.1 Definitions . 10
3.2 Abbreviations . 10
4 QoS methodology implementation . 11
4.1 General principles for the metric and indicator definition . 11
4.2 QoS requirements vs service element . 12
4.2.1 Flexibility . 12
4.2.2 Usability . 13
4.2.3 Security . 13
5 QoS metrics and indicators for the service behaviour in the service utilization . 13
5.1 Audio broadcast - Audiostreaming . 14
5.2 Directory enquiry services . 15
5.3 E-mail . 16
5.4 Fax . 18
5.5 Internet services . 19
5.6 Multimedia Message Service (MMS). 23
5.7 Operator services . 25
5.8 Short Message Service (SMS) . 26
5.9 Telephony . 28
5.10 Video broadcast - Video streaming . 31
5.11 Voice mail . 33
5.12 Voice messaging . 34
5.13 For further study . 36
6 QoS metrics and indicators for all steps of the customer relationship course other than utilization . 36
6.1 Sales . 36
6.1.1 Preliminary information . 36
6.1.1.1 Preliminary information needed by the customer for a telephony contract (fixed or mobile) . 38
6.1.1.2 Preliminary information needed by the customer for an ISP contract . 38
6.1.1.3 Preliminary information needed by the customer for a multi-services contract (fixed or mobile) . 39
6.1.2 Establishment of the contract (Terms and conditions) . 39
6.2 Service provisioning . 41
6.3 Service alteration and technical upgrade . 44
6.3.1 Service alteration . 44
6.3.2 Technical upgrade . 46
6.4 Service support . 47
6.4.1 Documentation . 48
6.4.2 Technical support . 49
6.4.3 Commercial support . 50
6.4.4 Complaint management . 52
6.5 Repair services . 54
6.6 Metering/charging/billing . 56
6.7 Network/service management by the customer . 59
6.8 Cessation . 60
7 Specific service elements . 62
ETSI

---------------------- Page: 3 ----------------------
4 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
7.1 Security as a service element . 62
7.1.1 Identification service element . 63
7.1.2 Authentication service element . 63
7.1.3 Authorization service element . 64
7.1.4 Encryption service element . 65
7.1.5 Non repudiation service element . 66
7.1.6 Time stamping service element . 67
7.1.7 Digital signature service element . 68
7.1.8 Certificate management service element . 68
7.2 Flexibility as a service element . 69
7.2.1 (Re)configuration service element . 70
7.2.2 (Re)provisionning service element . 70
8 Specific aspects of the general public users' criteria . 71
9 Conclusion . 72
Annex A: Bibliography . 73
History . 75

ETSI

---------------------- Page: 4 ----------------------
5 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This final draft ETSI Guide (EG) has been produced by ETSI User Group (USER), and is now submitted for the ETSI
standards Membership Approval Procedure.
It includes, among other contributions, excerpts of the final report of Bannock Consulting's project for the European
Commission's DG Information Society.
The present document is part 2 of a multi-part deliverable covering the quality of telecom services, as identified below:
Part 1: "Methodology for identification of indicators relevant to the Users";
Part 2: "User related indicators on a service specific basis";
Part 3: "Template for Service Level Agreements (SLA)".
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "may not", "need", "need not", "will",
"will not", "can" and "cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms
for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
Quality of Service can be evaluated from different perspectives and therefore using different measurement methods:
a) a first level of QoS is related to the reliability of the equipment and can be measured accurately via technical
means;
b) a second level is related to the service provisioning and is closely linked to the kind of use of the service.
Therefore appropriate criteria have to be defined according to this kind of use between the customer and the
supplier; the service delivery QoS depends on the network as well as on the server behaviours;
c) the last one is intended to measure the subjective satisfaction of the customer and there is often no other means
than a survey to get it (MOS value for media quality and OR for other services).
In the two first categories, technical means can be used to perform the measurements and in such cases, standards are
often useful to achieve a common approach; such standards are given as references where appropriate. They include a
precise definition of the relating metric and indicators and how to measure them. Unfortunately, they are not always
providing enough indications on the size and how to select the samples to be measured. ETSI EG 202 009-1 [i.11] can
help on these aspects but additional study may be needed to reach the right accuracy.
ETSI

---------------------- Page: 5 ----------------------
6 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
In the last category, the present document aims to give guidance on how to carry out the measurements including the
subjective ones.
Measurements of every interesting indicator all the time might be very expensive and can even jeopardize the network
and service performances. It can be more appropriate to get some of them via a poll on a limited number of users and
for a limited period of time. In addition, a third party may be needed to carry out these measurements to make them
more reliable and avoid any criticism from one of the involved parties.
ETSI

---------------------- Page: 6 ----------------------
7 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
1 Scope
In the current competitive world, Quality of Service (QoS) is becoming, jointly with cost, a key parameter in selling and
buying telecommunications services. At the same time, technology and liberalization trends are raising new types of
concerns unknown with the Plain Old Telephony Services (POTS) using switched connections provided by a single
monopoly supplier.
Nowadays, there are several standards describing QoS measurements but the question of which indicators are the most
interesting to be monitored from the users point of view and which values they should meet is still open. The present
document proposes a reference model to evaluate the Quality of Service from the users point of view, defining the
following concepts:
a) the appropriate indicators for a QoS assessment from the user point of view, i.e. KQI and SLO (see
ETSI EG 202 009-1 [i.11], clause 4.3);
b) the methods to acquire the indicator values (KQI) needed to assess the quality of service.
The main principles for these definitions are:
• To define the services according to the applications performed by the user and not by the technical solution:
for example, voice over IP is one of the many technical solutions to communicate between subscribers of the
world-wide telephone network; ATM, frame Relay, IP are some of the many technical solutions to ensure a
data transmission service between a terminal and a server or between networks. The quality criteria are the
same, only the Service Level achieved can be different.
• To define the quality criteria with respect of usage and not technique. In speech quality, users are more
interested in intelligibility than in bandwidth, distortion, signal to noise ratio or lost packets. Quality criteria
should be defined as SLO (KQI), then translated into technical criteria (KPI). This means that providers may
use different performance indicators to quantify and monitor the quality, depending on the technology used to
match a particular SLO.
Therefore, the present document does not intend to describe measurement techniques since several ETSI TCs are
dealing with such techniques and have the appropriate technical knowledge to develop standards in this area.
ETSI EG 202 009-1 [i.11] gives guidance in identifying the indicators relevant from the user requirement point of view.
If these metrics and indicators are used in a Service Level Agreements (SLA), it is crucial to define, at least for the most
important ones, the agreed quality targets. If the aim is a comparison of the respective providers' performances, then
quality targets can be provided as guidance for the general public to assess if the QoS of the results achieved is
satisfactory or not. The preferred values and the means to aggregate them are provided in a separate document
ETSI EG 202 934 [i.17].
The present document intends to define user related service specific KQI as far as possible using formal standards while
ETSI EG 202 009-3 [i.12] proposes a template for a SLA dealing with all service aspects, including penalties, escalation
procedures, areas of responsibility, etc. where these metrics and indicators can be used.
The purpose of the present document is to use the methodology described in ETSI EG 202 009-1 [i.11] to define, for
each QoS criterion, the relevant metrics and indicators (KQI) for a choice of services and for each step of the customer
relationship course. Hence each customer can have a comprehensive information on the features of the service he
intends to buy according to the various providers. This will enable him to select the best suited to his needs.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
reference document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
ETSI

---------------------- Page: 7 ----------------------
8 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] Recommendation ITU-T G.1010: "End-user multimedia QoS categories".
[i.2] Recommendation ITU-T P.800: "Methods for subjective determination of transmission quality".
[i.3] Recommendation ITU-T P.800.1: "Mean Opinion Score (MOS) terminology".
[i.4] Recommendation ITU-T P.862: "Perceptual evaluation of speech quality (PESQ), an objective
method for end-to-end speech quality assessment of narrowband telephone networks and speech
codecs".
[i.5] Recommendation ITU-T P.862.1: "Mapping function for transforming P.862 raw result scores to
MOS-LQO".
[i.6] Recommendation ITU-T P.862.2: "Wideband extension to Recommendation P.862 for the
assessment of wideband telephone networks and speech codecs".
[i.7] Recommendation ITU-T P.862.3: "Application guide for objective quality measurement based on
Recommendations P.862, P.862.1 and P.862.2".
[i.8] Recommendation ITU-T P.863: "Perceptual objective listening quality assessment".
[i.9] Recommendation ITU-T T.22: "Standardized test charts for document facsimile transmissions".
[i.10] ETSI EG 201 769: "Speech Processing, Transmission and Quality Aspects (STQ); QoS parameter
definitions and measurements; Parameters for voice telephony service required under the ONP
Voice Telephony Directive 98/10/EC".
[i.11] ETSI EG 202 009-1: "User Group; Quality of telecom services; Part 1: Methodology for
identification of parameters relevant to the Users".
[i.12] ETSI EG 202 009-3: "User Group; Quality of telecom services; Part 3: Template for Service Level
Agreements (SLA)".
[i.13] ETSI EG 202 057-2: "Speech and multimedia Transmission Quality (STQ); User related QoS
parameter definitions and measurements; Part 2: Voice telephony, Group 3 fax, modem data
services and SMS".
[i.14] ETSI EG 202 057-3: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 3: QoS parameters specific to Public Land
Mobile Networks (PLMN)".
[i.15] ETSI EG 202 057-4: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 4: Internet access".
[i.16] ETSI EG 202 843: "User Group; Quality of ICT services; Definitions and methods for assessing
the QoS parameters of the customer relationship stages other than utilization".
[i.17] ETSI EG 202 934: "User Group; The assessment of the overall Quality of Services (QoS) as
perceived by the users; Definition of QoS indexes for all the customer relationship stages".
[i.18] ETSI ES 202 057-1: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 1: General".
ETSI

---------------------- Page: 8 ----------------------
9 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
[i.19] ETSI ES 202 765-2: "Speech and multimedia Transmission Quality (STQ); QoS and network
performance metrics and measurement methods; Part 2: Transmission Quality Indicator combining
Voice Quality Metrics".
[i.20] ETSI ES 202 765-4: "Speech and multimedia Transmission Quality (STQ); QoS and network
performance metrics and measurement methods; Part 4: Indicators for supervision of Multiplay
services".
[i.21] ETSI TR 102 806: "User Group; Analysis of current End-to-End QoS standardization state".
[i.22] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
[i.23] ETSI TS 102 844: "User Group; Quality of Telecom Services; Conformity assessment;
Requirements for bodies providing QoS assessments and surveys".
[i.24] CEN/CENELEC/ETSI EN 301 549: "Accessibility requirements suitable for public procurement
of ICT products and services in Europe".
[i.25] CEN CWA14357: "CEN Workshop Agreement -Quality of Internet Service - Project Team Final
Report - ICS 35.240.60".
[i.26] IETF RFC 792: "Internet Control Message Protocol".
[i.27] ANSI/ASA S3.5-1997 (R2012): "American National Standard Methods for Calculation of the
Speech Intelligibility Index".
[i.28] Quality of Service Parameters for Internet Service Provision, final report of Bannock Consulting's
project for the European Commission's DG Information Society.
[i.29] ISO/IEC 18028-4:2005: "Information technology -- Security techniques -- IT network security --
Part 4: Securing remote access".
[i.30] ISO/IEC 7498-2: "Information processing systems -- Open Systems Interconnection -- Basic
Reference Model -- Part 2: Security Architecture".
[i.31] ISO/IEC 9797-1: "Information technology -- Security techniques -- Message Authentication Codes
(MACs) -- Part 1: Mechanisms using a block cipher".
[i.32] ISO/IEC 13888-1: "Information technology -- Security techniques -- Non-repudiation -- Part 1:
General".
[i.33] ISO/IEC 15945:2002: "Information technology -- Security techniques -- Specification of TTP
services to support the application of digital signatures".
[i.34] ISO/IEC 11770-3:1999: "Information technology -- Security techniques -- Key management --
Part 3: Mechanisms using asymmetric techniques".
[i.35] ITSEC: Information Technology Security Evaluation Criteria - Provisional Harmonized Criteria -
June 1991.
ETSI

---------------------- Page: 9 ----------------------
10 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the terms and definitions given in ETSI EG 202 009-1 [i.11] apply.
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ADSL Asymmetric Digital Subscriber Line
ATM Asynchroneous Transfer Mode
CRM Customer Relationship Management
DNS Domain Name System
FTP File Transfer Protocol
GPRS General Packet Radio Service
GSM Global System for Mobile
HTTP Hyper Text Transfer Protocol
ICMP Internet Control Management Protocol
ICT Information and Communication Technologies
IP Internet Protocol
IPTV Television Over Internet Protocol
ISDN Integrated Services Digital Network
ISP Internet Service Provider
IT Information Technology
ITSEC Information Technology SECurity
IVR Interactive Voice Response
KPI Key Performance Indicator
KQI Key Quality Indicator
MMS Multimedia Message Service
MMSC Multimedia Messaging Service Centre
MO Mobile Originate
MOS Mean Opinion Score
MT Mobile Terminate
OR Opinion Rating
PABX Private Automatic Branch eXchange
PDD Post Dialling Delay
PI Preliminary Information
PoP Point of Presence
POTS Plain Old Telephony Service
PSTN Public Switched Telephone Network
PTN Private Telecommunications Networks
QoS Quality of Service
RFC Request For Comment
SDS Short Data Service
SLA Service Level Agreement
SLO Service Level Objective
SMS Short Message Service
SP Service Provider
ST-MOS Listen Speech Quality Stability
TV Television
UE User equipment
UMTS Universal Mobile Telecommunications System
WGR WAP Get Request
ETSI

---------------------- Page: 10 ----------------------
11 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
4 QoS methodology implementation
To implement the methodology detailed in clause 6 of ETSI EG 202 009-1 [i.11], the process consists first in defining
carefully the intended service, including its conditions of use and service level range, then to check for each step of the
customer relationship course and each QoS criterion what are the best suited metrics and indicators to express the user's
requirements and to monitor these indicators appropriately. When this is done for each cell of the
ETSI EG 202 009-1 [i.11] matrix, there will probably be too much indicators for a convenient handling, therefore only
the most relevant should be taken for public
...