ETSI EG 202 009-2 V1.3.1 (2014-12)

User Group; Quality of telecom services; Part 2: User related indicators on a service specific basis

REG/USER-00042-2

General Information

• Status: Published
• Publication Date: 01-Dec-2014
• Technical Committee: USER - User Group
• Current Stage: 12 - Completion
• Due Date: 12-Dec-2014
• Completion Date: 02-Dec-2014

Standards Content (Sample)

Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)

ETSI GUIDE
User Group;
Quality of telecom services;
Part 2: User related indicators on a service specific basis

2 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)

Reference
REG/USER-00042-2
Keywords
QoS, quality, service, SLA, user
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2014.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 7
2 References . 7
2.1 Normative references . 8
2.2 Informative references . 8
3 Definitions and abbreviations . 10
3.1 Definitions . 10
3.2 Abbreviations . 10
4 QoS methodology implementation . 11
4.1 General principles for the metric and indicator definition . 11
4.2 QoS requirements vs service element . 12
4.2.1 Flexibility . 12
4.2.2 Usability . 13
4.2.3 Security . 13
5 QoS metrics and indicators for the service behaviour in the service utilization . 13
5.1 Audio broadcast - Audiostreaming . 14
5.2 Directory enquiry services . 15
5.3 E-mail . 16
5.4 Fax . 18
5.5 Internet services . 19
5.6 Multimedia Message Service (MMS). 23
5.7 Operator services . 25
5.8 Short Message Service (SMS) . 26
5.9 Telephony . 28
5.10 Video broadcast - Video streaming . 31
5.11 Voice mail . 33
5.12 Voice messaging . 34
5.13 For further study . 36
6 QoS metrics and indicators for all steps of the customer relationship course other than utilization . 36
6.1 Sales . 36
6.1.1 Preliminary information . 36
6.1.1.1 Preliminary information needed by the customer for a telephony contract (fixed or mobile) . 38
6.1.1.2 Preliminary information needed by the customer for an ISP contract . 38
6.1.1.3 Preliminary information needed by the customer for a multi-services contract (fixed or mobile) . 39
6.1.2 Establishment of the contract (Terms and conditions) . 39
6.2 Service provisioning . 41
6.3 Service alteration and technical upgrade . 44
6.3.1 Service alteration . 44
6.3.2 Technical upgrade . 46
6.4 Service support . 47
6.4.1 Documentation . 48
6.4.2 Technical support . 49
6.4.3 Commercial support . 50
6.4.4 Complaint management . 52
6.5 Repair services . 54
6.6 Metering/charging/billing . 56
6.7 Network/service management by the customer . 59
6.8 Cessation . 60
7 Specific service elements . 62
ETSI
4 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
7.1 Security as a service element . 62
7.1.1 Identification service element . 63
7.1.2 Authentication service element . 63
7.1.3 Authorization service element . 64
7.1.4 Encryption service element . 65
7.1.5 Non repudiation service element . 66
7.1.6 Time stamping service element . 67
7.1.7 Digital signature service element . 68
7.1.8 Certificate management service element . 68
7.2 Flexibility as a service element . 69
7.2.1 (Re)configuration service element . 70
7.2.2 (Re)provisionning service element . 70
8 Specific aspects of the general public users' criteria . 71
9 Conclusion . 72
Annex A: Bibliography . 73
History . 75

ETSI
5 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This final draft ETSI Guide (EG) has been produced by ETSI User Group (USER), and is now submitted for the ETSI
standards Membership Approval Procedure.
It includes, among other contributions, excerpts of the final report of Bannock Consulting's project for the European
Commission's DG Information Society.
The present document is part 2 of a multi-part deliverable covering the quality of telecom services, as identified below:
Part 1: "Methodology for identification of indicators relevant to the Users";
Part 2: "User related indicators on a service specific basis";
Part 3: "Template for Service Level Agreements (SLA)".
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "may not", "need", "need not", "will",
"will not", "can" and "cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms
for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
Quality of Service can be evaluated from different perspectives and therefore using different measurement methods:
a) a first level of QoS is related to the reliability of the equipment and can be measured accurately via technical
means;
b) a second level is related to the service provisioning and is closely linked to the kind of use of the service.
Therefore appropriate criteria have to be defined according to this kind of use between the customer and the
supplier; the service delivery QoS depends on the network as well as on the server behaviours;
c) the last one is intended to measure the subjective satisfaction of the customer and there is often no other means
than a survey to get it (MOS value for media quality and OR for other services).
In the two first categories, technical means can be used to perform the measurements and in such cases, standards are
often useful to achieve a common approach; such standards are given as references where appropriate. They include a
precise definition of the relating metric and indicators and how to measure them. Unfortunately, they are not always
providing enough indications on the size and how to select the samples to be measured. ETSI EG 202 009-1 [i.11] can
help on these aspects but additional study may be needed to reach the right accuracy.
ETSI
6 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
In the last category, the present document aims to give guidance on how to carry out the measurements including the
subjective ones.
Measurements of every interesting indicator all the time might be very expensive and can even jeopardize the network
and service performances. It can be more appropriate to get some of them via a poll on a limited number of users and
for a limited period of time. In addition, a third party may be needed to carry out these measurements to make them
more reliable and avoid any criticism from one of the involved parties.
ETSI
7 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
1 Scope
In the current competitive world, Quality of Service (QoS) is becoming, jointly with cost, a key parameter in selling and
buying telecommunications services. At the same time, technology and liberalization trends are raising new types of
concerns unknown with the Plain Old Telephony Services (POTS) using switched connections provided by a single
monopoly supplier.
Nowadays, there are several standards describing QoS measurements but the question of which indicators are the most
interesting to be monitored from the users point of view and which values they should meet is still open. The present
document proposes a reference model to evaluate the Quality of Service from the users point of view, defining the
following concepts:
a) the appropriate indicators for a QoS assessment from the user point of view, i.e. KQI and SLO (see
ETSI EG 202 009-1 [i.11], clause 4.3);
b) the methods to acquire the indicator values (KQI) needed to assess the quality of service.
The main principles for these definitions are:
• To define the services according to the applications performed by the user and not by the technical solution:
for example, voice over IP is one of the many technical solutions to communicate between subscribers of the
world-wide telephone network; ATM, frame Relay, IP are some of the many technical solutions to ensure a
data transmission service between a terminal and a server or between networks. The quality criteria are the
same, only the Service Level achieved can be different.
• To define the quality criteria with respect of usage and not technique. In speech quality, users are more
interested in intelligibility than in bandwidth, distortion, signal to noise ratio or lost packets. Quality criteria
should be defined as SLO (KQI), then translated into technical criteria (KPI). This means that providers may
use different performance indicators to quantify and monitor the quality, depending on the technology used to
match a particular SLO.
Therefore, the present document does not intend to describe measurement techniques since several ETSI TCs are
dealing with such techniques and have the appropriate technical knowledge to develop standards in this area.
ETSI EG 202 009-1 [i.11] gives guidance in identifying the indicators relevant from the user requirement point of view.
If these metrics and indicators are used in a Service Level Agreements (SLA), it is crucial to define, at least for the most
important ones, the agreed quality targets. If the aim is a comparison of the respective providers' performances, then
quality targets can be provided as guidance for the general public to assess if the QoS of the results achieved is
satisfactory or not. The preferred values and the means to aggregate them are provided in a separate document
ETSI EG 202 934 [i.17].
The present document intends to define user related service specific KQI as far as possible using formal standards while
ETSI EG 202 009-3 [i.12] proposes a template for a SLA dealing with all service aspects, including penalties, escalation
procedures, areas of responsibility, etc. where these metrics and indicators can be used.
The purpose of the present document is to use the methodology described in ETSI EG 202 009-1 [i.11] to define, for
each QoS criterion, the relevant metrics and indicators (KQI) for a choice of services and for each step of the customer
relationship course. Hence each customer can have a comprehensive information on the features of the service he
intends to buy according to the various providers. This will enable him to select the best suited to his needs.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
reference document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
ETSI
8 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] Recommendation ITU-T G.1010: "End-user multimedia QoS categories".
[i.2] Recommendation ITU-T P.800: "Methods for subjective determination of transmission quality".
[i.3] Recommendation ITU-T P.800.1: "Mean Opinion Score (MOS) terminology".
[i.4] Recommendation ITU-T P.862: "Perceptual evaluation of speech quality (PESQ), an objective
method for end-to-end speech quality assessment of narrowband telephone networks and speech
codecs".
[i.5] Recommendation ITU-T P.862.1: "Mapping function for transforming P.862 raw result scores to
MOS-LQO".
[i.6] Recommendation ITU-T P.862.2: "Wideband extension to Recommendation P.862 for the
assessment of wideband telephone networks and speech codecs".
[i.7] Recommendation ITU-T P.862.3: "Application guide for objective quality measurement based on
Recommendations P.862, P.862.1 and P.862.2".
[i.8] Recommendation ITU-T P.863: "Perceptual objective listening quality assessment".
[i.9] Recommendation ITU-T T.22: "Standardized test charts for document facsimile transmissions".
[i.10] ETSI EG 201 769: "Speech Processing, Transmission and Quality Aspects (STQ); QoS parameter
definitions and measurements; Parameters for voice telephony service required under the ONP
Voice Telephony Directive 98/10/EC".
[i.11] ETSI EG 202 009-1: "User Group; Quality of telecom services; Part 1: Methodology for
identification of parameters relevant to the Users".
[i.12] ETSI EG 202 009-3: "User Group; Quality of telecom services; Part 3: Template for Service Level
Agreements (SLA)".
[i.13] ETSI EG 202 057-2: "Speech and multimedia Transmission Quality (STQ); User related QoS
parameter definitions and measurements; Part 2: Voice telephony, Group 3 fax, modem data
services and SMS".
[i.14] ETSI EG 202 057-3: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 3: QoS parameters specific to Public Land
Mobile Networks (PLMN)".
[i.15] ETSI EG 202 057-4: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 4: Internet access".
[i.16] ETSI EG 202 843: "User Group; Quality of ICT services; Definitions and methods for assessing
the QoS parameters of the customer relationship stages other than utilization".
[i.17] ETSI EG 202 934: "User Group; The assessment of the overall Quality of Services (QoS) as
perceived by the users; Definition of QoS indexes for all the customer relationship stages".
[i.18] ETSI ES 202 057-1: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 1: General".
ETSI
9 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
[i.19] ETSI ES 202 765-2: "Speech and multimedia Transmission Quality (STQ); QoS and network
performance metrics and measurement methods; Part 2: Transmission Quality Indicator combining
Voice Quality Metrics".
[i.20] ETSI ES 202 765-4: "Speech and multimedia Transmission Quality (STQ); QoS and network
performance metrics and measurement methods; Part 4: Indicators for supervision of Multiplay
services".
[i.21] ETSI TR 102 806: "User Group; Analysis of current End-to-End QoS standardization state".
[i.22] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
[i.23] ETSI TS 102 844: "User Group; Quality of Telecom Services; Conformity assessment;
Requirements for bodies providing QoS assessments and surveys".
[i.24] CEN/CENELEC/ETSI EN 301 549: "Accessibility requirements suitable for public procurement
of ICT products and services in Europe".
[i.25] CEN CWA14357: "CEN Workshop Agreement -Quality of Internet Service - Project Team Final
Report - ICS 35.240.60".
[i.26] IETF RFC 792: "Internet Control Message Protocol".
[i.27] ANSI/ASA S3.5-1997 (R2012): "American National Standard Methods for Calculation of the
Speech Intelligibility Index".
[i.28] Quality of Service Parameters for Internet Service Provision, final report of Bannock Consulting's
project for the European Commission's DG Information Society.
[i.29] ISO/IEC 18028-4:2005: "Information technology -- Security techniques -- IT network security --
Part 4: Securing remote access".
[i.30] ISO/IEC 7498-2: "Information processing systems -- Open Systems Interconnection -- Basic
Reference Model -- Part 2: Security Architecture".
[i.31] ISO/IEC 9797-1: "Information technology -- Security techniques -- Message Authentication Codes
(MACs) -- Part 1: Mechanisms using a block cipher".
[i.32] ISO/IEC 13888-1: "Information technology -- Security techniques -- Non-repudiation -- Part 1:
General".
[i.33] ISO/IEC 15945:2002: "Information technology -- Security techniques -- Specification of TTP
services to support the application of digital signatures".
[i.34] ISO/IEC 11770-3:1999: "Information technology -- Security techniques -- Key management --
Part 3: Mechanisms using asymmetric techniques".
[i.35] ITSEC: Information Technology Security Evaluation Criteria - Provisional Harmonized Criteria -
June 1991.
ETSI
10 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the terms and definitions given in ETSI EG 202 009-1 [i.11] apply.
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ADSL Asymmetric Digital Subscriber Line
ATM Asynchroneous Transfer Mode
CRM Customer Relationship Management
DNS Domain Name System
FTP File Transfer Protocol
GPRS General Packet Radio Service
GSM Global System for Mobile
HTTP Hyper Text Transfer Protocol
ICMP Internet Control Management Protocol
ICT Information and Communication Technologies
IP Internet Protocol
IPTV Television Over Internet Protocol
ISDN Integrated Services Digital Network
ISP Internet Service Provider
IT Information Technology
ITSEC Information Technology SECurity
IVR Interactive Voice Response
KPI Key Performance Indicator
KQI Key Quality Indicator
MMS Multimedia Message Service
MMSC Multimedia Messaging Service Centre
MO Mobile Originate
MOS Mean Opinion Score
MT Mobile Terminate
OR Opinion Rating
PABX Private Automatic Branch eXchange
PDD Post Dialling Delay
PI Preliminary Information
PoP Point of Presence
POTS Plain Old Telephony Service
PSTN Public Switched Telephone Network
PTN Private Telecommunications Networks
QoS Quality of Service
RFC Request For Comment
SDS Short Data Service
SLA Service Level Agreement
SLO Service Level Objective
SMS Short Message Service
SP Service Provider
ST-MOS Listen Speech Quality Stability
TV Television
UE User equipment
UMTS Universal Mobile Telecommunications System
WGR WAP Get Request
ETSI
11 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
4 QoS methodology implementation
To implement the methodology detailed in clause 6 of ETSI EG 202 009-1 [i.11], the process consists first in defining
carefully the intended service, including its conditions of use and service level range, then to check for each step of the
customer relationship course and each QoS criterion what are the best suited metrics and indicators to express the user's
requirements and to monitor these indicators appropriately. When this is done for each cell of the
ETSI EG 202 009-1 [i.11] matrix, there will probably be too much indicators for a convenient handling, therefore only
the most relevant should be taken for publication or to include in a SLA.
Trying to limit drastically the number of indicators may be counter-productive as it gives the provider an incentive to
focus on a particular measure, perhaps at the expense of the general QoS. Also trade-offs may be necessary - for
example between cost and reliability. Benchmarking, when available, can provide useful support in this aspect. To
ensure the best compromise between the number of indicators and their ability to provide an effective assessment of the
QoS, it would be helpful to use surveys focusing on users' complaints to identify where are the main non quality issues
to adapt the indicator sample accordingly. This indicator selection should be reviewed regularly.
4.1 General principles for the metric and indicator definition
Every technical measurement should keep in with the customer perception (e.g. end-to-end transit time), although
suppliers may need to carry out technical measurements (KPI) on particular points to ensure a good customer perceived
quality even if these KPI are not directly perceptible to the user.
Some general principles should apply:
• Beware of mean values that gives figures which might be very far from a particular customer feeling.
• Focus on disturbance (should be 0) rather than performance (close to 100 %) even if it is only a presentation
issue.
• Use figures that may be aggregated (disturbance rate).
• For QoS targets define thresholds suited to the user perceived QoS.
When reading the present document, anyone should have in mind that a KQI is, in the user language, the translation of a
rate, a frequency or any other KPI, in a tangible perception from the user viewpoint. In this context, a KQI may involve
more than one KPI. The tables provided in the following clauses endeavour to give guidance on the relevant metrics and
indicators for the main electronic communication services and, when available, the corresponding standards.
Nevertheless, it should be clear that users can ask for different quality levels for the same service used in different
contexts. It is crucial to notice that QoS should always be assessed with regard to thresholds expected matching the
current user satisfaction in order to avoid a race to useless performances. Such QoS thresholds may be defined on a case
by case basis with, possibly, different requirements for different uses. In any case, the publication of KQI values is
expected to help to the users' freedom of choice, making them able to make an informed choice.
KQI for the service behaviour in the service utilization are in many cases service specific while in the other steps of the
customer relationship course they are often common to most services. Therefore to ease the reading, the definitions of
QoS metrics and indicators are split in two parts: those related to the operational aspects (Service utilization) and those
related to the other aspects of the service: sales, provision, alteration, upgrade, commercial and technical support,
complaint management, repair, charging/billing, network/service management, cessation.
It is important that the following points are made:
• What are the KQI that are pertinent to the particular service?
• How are these to be assessed?
• Who will assess them?
• What are the acceptable assessment procedures (test specification, i.e. ITU-T recommendation, ETSI standard
or survey and the frequency of measurement, sample size, confidence limits, etc.)?
• What are the acceptable, nominal service level ranges?
ETSI
12 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
ETSI EG 202 009-1 [i.11] provides guidance on these aspects that have to be specified for any QoS assessment. In the
following clauses, tables are given to define the relevant QoS metrics and indicators for a selection of services along the
various aspects of the ICT customer relationship course.
Indicators may be assessed by various means: technical measurements performed by the supplier or an independent
organization, or a poll of a user panel.
According to users' opinion, in most cases both technical measurements and surveys among users are useful to achieve a
realistic assessment of the QoS.
Along with these considerations, the tables in the following clauses will consider various metrics for the QoS, seeking
to identify a set of measures that are expected to form the basis for judging the performance of the supplier from the
point of view of the consumers or business users. In this area the statistical quality is crucial to the credibility of the
results and should be given for most if not all of these measurements.
In these tables, the metric gives an expression of a criterion from the user viewpoint and relevant to his dashboard.
Indicators are used to give a quantifiable value allowing for the appraisal of the quality of a given service.
These tables are built on the basis of two general principles:
1) It is crucial, to avoid misunderstanding, that the metric and indicators to measure the QoS can be used and
managed by both the users and the providers, even if these metrics are viewed from different perspectives.
2) Theoretically, at least one KPI is necessary for an actual evaluation of each criterion and all the criteria are
needed for a comprehensive QoS appraisal of a given service aspect. The context for KQI is somewhat
different as a KQI may involve more than one criterion. The tables are built on the principle that all the criteria
are involved in at least one KQI. Nevertheless, as explained earlier, for practical reasons, a reduced set of
carefully selected indicators may be used for an efficient QoS monitoring.
Hence, each table shows, for each KQI, the metrics and the related indicator definitions. In addition, for each indicator,
the last column shows which type of measurement is appropriate.
As indicated earlier in this clause, the indicators given in these tables often refer to standards that provide additional
information about how the measurements should be performed and who is expected to perform and provide them. Any
one intending to assess QoS is invited to carefully read them. When there is no standard available to define appropriate
metrics and/or indicators, definitions are nevertheless proposed to compensate for the lack of such standard.
Since the previous version, a lot of efforts have been made to make available new standards for the assessment of KQI.
Not necessarily all the indicators defined in the standards are listed in clauses 5 and 6 tables but only those considered
as the most relevant to the users. In addition, it is to the organization using the present document to choose among these
indicators those that are appropriate to the particular situation to assess.
4.2 QoS requirements vs service element
Since nowadays, most services are in fact a service make-up, it is crucial before intending to assess the QoS of a service
(behaviour), to describe what the service is intended for, what are the conditions of use, what are the service elements
included, etc. In particular, for the GSM, UMTS, GPRS and ADSL services, the areas where the service is offered
should be specified.
Therefore, since the tables dealing with the indicators for the technical quality are service specific, the definition of the
functionality of the intended service is given at the beginning of the table.
In some cases, options can be bought to gain extended service features, e.g. flexible performances, high availability, etc.
These particular issues should be considered as service elements which can have their own QoS checking process and
might even impact the overall QoS. The QoS assessment of these specific service elements are tackled in the next
clauses.
4.2.1 Flexibility
Once the contract signed, it can be more or less easy to adjust some service settings according to the evolution of the
user needs without change in the main features of the service. This is why, according to the matrix provided in table 2
of clause 6.1 of ETSI EG 202 009-1 [i.11], such aspect of the flexibility is considered as a QoS requirement to be
assessed for each customer relationship step.
ETSI
13 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
Nevertheless, in some other cases, the customer may wish to have the service features matching in real time the
requirements of particular type of use of the service. In such case, the flexibility can be considered as a service element
and therefore its QoS having to be analyzed with its own QoS checking process i.e. with regard to the criteria of
availability, integrity, time, capacity, etc.
Clause 7.2 is dedicated to the flexibility as service elements related to the service utilization, such as (re)configuration
and (re)provisioning.
4.2.2 Usability
Usability represents effectiveness, efficiency and satisfaction with which specified users can achieve specified goals
(tasks) in a particular environment with respect to the user profile.
As a first step the usability is considered as a QoS requirement to be assessed for each customer relationship step.
In a second time service elements such as transcoder, web browser could be defined.
4.2.3 Security
Security, data protection, privacy are key user concerns that have to be assessed for each customer relationship step. In
this context, according to ITSEC [i.35] IT security embraces multiple aspects that are detailed in clause 6 of Part 1 with
several service elements to achieve each of them.
Since, according to the matrix in table 2 of clause 6.1 of ETSI EG 202 009-1 [i.11], the present document is intended to
identify how the means available to assess the KPI can help to check that the provider contractual commitments are
met, all the security sections in clauses 5 and 6 are endeavouring to identify the KQI related to the key security
objectives defined according to the guidances provided in clause 6.1 of ETSI EG 202 009-1 [i.11]. In addition, if
needed, security can be assessed as a particular service element with regard to the QoS criteria and its impact on the
overall QoS of any step of the customer relationship course but particularly the operational ones. The clause 7.1 is
dedicated to the specific QoS criteria related to this security service element.
5 QoS metrics and indicators for the service behaviour
in the service utilization
It is very important to notice that the tables in this clause are intended to be focused on the services and to be
technology agnostic: being user oriented, the metric should be the same whatever the technology of the bearer network.
The differences due to differing technologies are expected to appear in the results, the presentation of them taking
possibly account of the communication path. These does not mean that a breakdown of the results according to the
technology or to the type of contract is irrelevant. Therefore, when the QoS assessment is expected to provide for a
comparison between the QoS achieved by several providers, a particular care should be given to the composition of the
test sample for each of them. Particularly when IP technology is used to provide the service, location, type of contract
and specification of the set-top-box of each sample item should be carefully selected to ensure the measurements
actually represent the QoS provided.
The services listed here are drawn from Recommendation ITU-T G.1010 [i.1] with some changes to take into account
the current market situation. The definition of the function achieved by each of them is given in the first sentence of the
related clause. Some services have usually several components and, when appropriate, indicators for each of such
components are identified separately in the list of the service indicators. Due to time and resource constraints, QoS
metrics and indicators are not available yet for all these services. As mentioned in clause 5.13, other services could be
provided in a later edition.
The methodology can be used to monitor the compliance of a provision to the provider commitments as well as a
performance assessment without any reference, for example in the intention to provide comparative information to a
prospect in order to help him to choose the offer best suited to his wishes.
It is worth noting that the reference to formal standards is crucial to the reliability of the results and to enable
comparisons between providers. On the other hand, the targets to meet are related to a contractual approach between the
provider and the regulator or between the provider and his customers.
ETSI
14 Final draft ETSI EG 202 009-2 V1.3.0 (2014-09)
What is at stake is that the user can know and possibly negotiate the targets on which the provider is committed and
then to get access to the information enabling him to check whether these commitments are met.
Users should be interested to have some guidance on which targets KQI should reach to ensure a good QoS but at this
stage of the work, it was not possible to provide such guidance. This will be for a later revision if the QoS measurement
experience has grown enough.
5.1 Audio broadcast - Audiostreaming
Audio broadcast/streaming: A mechanism whereby audio content can be rendered at the same time that it is being
transmitted to the client over the data network.
NOTE: The indicator evaluates the quality of audio stream.
In the context of end user evaluation without access to provider network, the assessment method should
perform a methodology "without reference".
Speech evaluation methodologies or speech quality assessment models are not appropriate to evaluate
audio quality. These methodologies or these models cannot be performed for audio quality evaluation.
In this context the most suitable approach is to determine "audio cut" (or "audio lack") occurrence and
low audio level sequence in the audio stream.
This analysis should be performed on the left and right channels, when available.
This indicator could be presented as the number of degradation or in ratio (number of degradation by time
unit).
An alternative assessment of the audio quality may be based on subjective tests or surveys by a
representative user panel (MOS value).
Availability
Metric Related indicator definition (KQI)
Rate of server accessibility Percentage of successful log-ins with respect to the total number of
attempts,
{Reference: Successful log-in ratio; ETSI EG 202 057-4 [i.15]}

Integrity
Metric Related indicator definition (KQI)
Listening audio loss ratio Consolidated duration of audible audio cut over one minute listening.
{Reference: Audio quality; ETSI ES 202 765-4 [i.20], clause 8.4}
Listening audio loss frequency Number of audible audio cut over one minute listening.
{Reference: Audio
...

ETSI GUIDE
User Group;
Quality of telecom services;
Part 2: User related indicators on a service specific basis

2 ETSI EG 202 009-2 V1.3.1 (2014-12)

Reference
REG/USER-00042-2
Keywords
QoS, quality, service, SLA, user
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2014.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 ETSI EG 202 009-2 V1.3.1 (2014-12)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 7
2 References . 7
2.1 Normative references . 8
2.2 Informative references . 8
3 Definitions and abbreviations . 10
3.1 Definitions . 10
3.2 Abbreviations . 10
4 QoS methodology implementation . 11
4.1 General principles for the metric and indicator definition . 11
4.2 QoS requirements vs service element . 12
4.2.1 Flexibility . 12
4.2.2 Usability . 13
4.2.3 Security . 13
5 QoS metrics and indicators for the service behaviour in the service utilization . 13
5.1 Audio broadcast - Audiostreaming . 14
5.2 Directory enquiry services . 15
5.3 E-mail . 16
5.4 Fax . 18
5.5 Internet services . 19
5.6 Multimedia Message Service (MMS). 23
5.7 Operator services . 25
5.8 Short Message Service (SMS) . 26
5.9 Telephony . 28
5.10 Video broadcast - Video streaming . 31
5.11 Voice mail . 33
5.12 Voice messaging . 34
5.13 For further study . 35
6 QoS metrics and indicators for all steps of the customer relationship course other than utilization . 36
6.1 Sales . 36
6.1.1 Preliminary information . 36
6.1.1.1 Preliminary information needed by the customer for a telephony contract (fixed or mobile) . 38
6.1.1.2 Preliminary information needed by the customer for an ISP contract . 38
6.1.1.3 Preliminary information needed by the customer for a multi-services contract (fixed or mobile) . 39
6.1.2 Establishment of the contract (Terms and conditions) . 39
6.2 Service provisioning . 41
6.3 Service alteration and technical upgrade . 44
6.3.1 Service alteration . 44
6.3.2 Technical upgrade . 46
6.4 Service support . 47
6.4.1 Documentation . 48
6.4.2 Technical support . 49
6.4.3 Commercial support . 50
6.4.4 Complaint management . 52
6.5 Repair services . 54
6.6 Metering/charging/billing . 56
6.7 Network/service management by the customer . 59
6.8 Cessation . 61
7 Specific service elements . 62
ETSI
4 ETSI EG 202 009-2 V1.3.1 (2014-12)
7.1 Security as a service element . 62
7.1.1 Identification service element . 63
7.1.2 Authentication service element . 64
7.1.3 Authorization service element . 64
7.1.4 Encryption service element . 65
7.1.5 Non repudiation service element . 66
7.1.6 Time stamping service element . 67
7.1.7 Digital signature service element . 68
7.1.8 Certificate management service element . 68
7.2 Flexibility as a service element . 69
7.2.1 (Re)configuration service element . 70
7.2.2 (Re)provisionning service element . 70
8 Specific aspects of the general public users' criteria . 71
9 Conclusion . 72
Annex A: Bibliography . 73
History . 75

ETSI
5 ETSI EG 202 009-2 V1.3.1 (2014-12)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI User Group (USER).
It includes, among other contributions, excerpts of the final report of Bannock Consulting's project for the European
Commission's DG Information Society.
The present document is part 2 of a multi-part deliverable covering the quality of telecom services, as identified below:
Part 1: "Methodology for identification of indicators relevant to the Users";
Part 2: "User related indicators on a service specific basis";
Part 3: "Template for Service Level Agreements (SLA)".
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "may not", "need", "need not", "will",
"will not", "can" and "cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms
for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
Quality of Service can be evaluated from different perspectives and therefore using different measurement methods:
a) a first level of QoS is related to the reliability of the equipment and can be measured accurately via technical
means;
b) a second level is related to the service provisioning and is closely linked to the kind of use of the service.
Therefore appropriate criteria have to be defined according to this kind of use between the customer and the
supplier; the service delivery QoS depends on the network as well as on the server behaviours;
c) the last one is intended to measure the subjective satisfaction of the customer and there is often no other means
than a survey to get it (MOS value for media quality and OR for other services).
In the two first categories, technical means can be used to perform the measurements and in such cases, standards are
often useful to achieve a common approach; such standards are given as references where appropriate. They include a
precise definition of the relating metric and indicators and how to measure them. Unfortunately, they are not always
providing enough indications on the size and how to select the samples to be measured. ETSI EG 202 009-1 [i.11] can
help on these aspects but additional study may be needed to reach the right accuracy.
In the last category, the present document aims to give guidance on how to carry out the measurements including the
subjective ones.
ETSI
6 ETSI EG 202 009-2 V1.3.1 (2014-12)
Measurements of every interesting indicator all the time might be very expensive and can even jeopardize the network
and service performances. It can be more appropriate to get some of them via a poll on a limited number of users and
for a limited period of time. In addition, a third party may be needed to carry out these measurements to make them
more reliable and avoid any criticism from one of the involved parties.
ETSI
7 ETSI EG 202 009-2 V1.3.1 (2014-12)
1 Scope
In the current competitive world, Quality of Service (QoS) is becoming, jointly with cost, a key parameter in selling and
buying telecommunications services. At the same time, technology and liberalization trends are raising new types of
concerns unknown with the Plain Old Telephony Services (POTS) using switched connections provided by a single
monopoly supplier.
Nowadays, there are several standards describing QoS measurements but the question of which indicators are the most
interesting to be monitored from the users point of view and which values they should meet is still open. The present
document proposes a reference model to evaluate the Quality of Service from the users point of view, defining the
following concepts:
a) the appropriate indicators for a QoS assessment from the user point of view, i.e. KQI and SLO (see ETSI
EG 202 009-1 [i.11], clause 4.3);
b) the methods to acquire the indicator values (KQI) needed to assess the quality of service.
The main principles for these definitions are:
• To define the services according to the applications performed by the user and not by the technical solution:
for example, voice over IP is one of the many technical solutions to communicate between subscribers of the
world-wide telephone network; ATM, frame Relay, IP are some of the many technical solutions to ensure a
data transmission service between a terminal and a server or between networks. The quality criteria are the
same, only the Service Level achieved can be different.
• To define the quality criteria with respect of usage and not technique. In speech quality, users are more
interested in intelligibility than in bandwidth, distortion, signal to noise ratio or lost packets. Quality criteria
should be defined as SLO (KQI), then translated into technical criteria (KPI). This means that providers may
use different performance indicators to quantify and monitor the quality, depending on the technology used to
match a particular SLO.
Therefore, the present document does not intend to describe measurement techniques since several ETSI TCs are
dealing with such techniques and have the appropriate technical knowledge to develop standards in this area.
ETSI EG 202 009-1 [i.11] gives guidance in identifying the indicators relevant from the user requirement point of view.
If these metrics and indicators are used in a Service Level Agreements (SLA), it is crucial to define, at least for the most
important ones, the agreed quality targets. If the aim is a comparison of the respective providers' performances, then
quality targets can be provided as guidance for the general public to assess if the QoS of the results achieved is
satisfactory or not. The preferred values and the means to aggregate them are provided in a separate document
ETSI EG 202 934 [i.17].
The present document intends to define user related service specific KQI as far as possible using formal standards while
ETSI EG 202 009-3 [i.12] proposes a template for a SLA dealing with all service aspects, including penalties, escalation
procedures, areas of responsibility, etc. where these metrics and indicators can be used.
The purpose of the present document is to use the methodology described in ETSI EG 202 009-1 [i.11] to define, for
each QoS criterion, the relevant metrics and indicators (KQI) for a choice of services and for each step of the customer
relationship course. Hence each customer can have a comprehensive information on the features of the service he
intends to buy according to the various providers. This will enable him to select the best suited to his needs.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
reference document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
ETSI
8 ETSI EG 202 009-2 V1.3.1 (2014-12)
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] Recommendation ITU-T G.1010: "End-user multimedia QoS categories".
[i.2] Recommendation ITU-T P.800: "Methods for subjective determination of transmission quality".
[i.3] Recommendation ITU-T P.800.1: "Mean Opinion Score (MOS) terminology".
[i.4] Recommendation ITU-T P.862: "Perceptual evaluation of speech quality (PESQ), an objective
method for end-to-end speech quality assessment of narrowband telephone networks and speech
codecs".
[i.5] Recommendation ITU-T P.862.1: "Mapping function for transforming P.862 raw result scores to
MOS-LQO".
[i.6] Recommendation ITU-T P.862.2: "Wideband extension to Recommendation P.862 for the
assessment of wideband telephone networks and speech codecs".
[i.7] Recommendation ITU-T P.862.3: "Application guide for objective quality measurement based on
Recommendations P.862, P.862.1 and P.862.2".
[i.8] Recommendation ITU-T P.863: "Perceptual objective listening quality assessment".
[i.9] Recommendation ITU-T T.22: "Standardized test charts for document facsimile transmissions".
[i.10] ETSI EG 201 769: "Speech Processing, Transmission and Quality Aspects (STQ); QoS parameter
definitions and measurements; Parameters for voice telephony service required under the ONP
Voice Telephony Directive 98/10/EC".
[i.11] ETSI EG 202 009-1: "User Group; Quality of telecom services; Part 1: Methodology for
identification of parameters relevant to the Users".
[i.12] ETSI EG 202 009-3: "User Group; Quality of telecom services; Part 3: Template for Service Level
Agreements (SLA)".
[i.13] ETSI EG 202 057-2: "Speech and multimedia Transmission Quality (STQ); User related QoS
parameter definitions and measurements; Part 2: Voice telephony, Group 3 fax, modem data
services and SMS".
[i.14] ETSI EG 202 057-3: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 3: QoS parameters specific to Public Land
Mobile Networks (PLMN)".
[i.15] ETSI EG 202 057-4: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 4: Internet access".
[i.16] ETSI EG 202 843: "User Group; Quality of ICT services; Definitions and methods for assessing
the QoS parameters of the customer relationship stages other than utilization".
[i.17] ETSI EG 202 934: "User Group; The assessment of the overall Quality of Services (QoS) as
perceived by the users; Definition of QoS indexes for all the customer relationship stages".
[i.18] ETSI ES 202 057-1: "Speech Processing, Transmission and Quality Aspects (STQ); User related
QoS parameter definitions and measurements; Part 1: General".
ETSI
9 ETSI EG 202 009-2 V1.3.1 (2014-12)
[i.19] ETSI ES 202 765-2: "Speech and multimedia Transmission Quality (STQ); QoS and network
performance metrics and measurement methods; Part 2: Transmission Quality Indicator combining
Voice Quality Metrics".
[i.20] ETSI ES 202 765-4: "Speech and multimedia Transmission Quality (STQ); QoS and network
performance metrics and measurement methods; Part 4: Indicators for supervision of Multiplay
services".
[i.21] ETSI TR 102 806: "User Group; Analysis of current End-to-End QoS standardization state".
[i.22] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
[i.23] ETSI TS 102 844: "User Group; Quality of Telecom Services; Conformity assessment;
Requirements for bodies providing QoS assessments and surveys".
[i.24] CEN/CENELEC/ETSI EN 301 549: "Accessibility requirements suitable for public procurement
of ICT products and services in Europe".
[i.25] CEN CWA14357: "CEN Workshop Agreement -Quality of Internet Service - Project Team Final
Report - ICS 35.240.60".
[i.26] IETF RFC 792: "Internet Control Message Protocol".
[i.27] ANSI/ASA S3.5-1997 (R2012): "American National Standard Methods for Calculation of the
Speech Intelligibility Index".
[i.28] Quality of Service Parameters for Internet Service Provision, final report of Bannock Consulting's
project for the European Commission's DG Information Society.
[i.29] ISO/IEC 18028-4:2005: "Information technology -- Security techniques -- IT network security --
Part 4: Securing remote access".
[i.30] ISO/IEC 7498-2: "Information processing systems -- Open Systems Interconnection -- Basic
Reference Model -- Part 2: Security Architecture".
[i.31] ISO/IEC 9797-1: "Information technology -- Security techniques -- Message Authentication Codes
(MACs) -- Part 1: Mechanisms using a block cipher".
[i.32] ISO/IEC 13888-1: "Information technology -- Security techniques -- Non-repudiation -- Part 1:
General".
[i.33] ISO/IEC 15945:2002: "Information technology -- Security techniques -- Specification of TTP
services to support the application of digital signatures".
[i.34] ISO/IEC 11770-3:1999: "Information technology -- Security techniques -- Key management --
Part 3: Mechanisms using asymmetric techniques".
[i.35] ITSEC: "Information Technology Security Evaluation Criteria - Provisional Harmonized Criteria"
June 1991.
ETSI
10 ETSI EG 202 009-2 V1.3.1 (2014-12)
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the terms and definitions given in ETSI EG 202 009-1 [i.11] apply.
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ADSL Asymmetric Digital Subscriber Line
ATM Asynchroneous Transfer Mode
CRM Customer Relationship Management
DNS Domain Name System
FTP File Transfer Protocol
GPRS General Packet Radio Service
GSM Global System for Mobile
HTTP Hyper Text Transfer Protocol
ICMP Internet Control Management Protocol
ICT Information and Communication Technologies
IP Internet Protocol
IPTV Television over Internet Protocol
ISDN Integrated Services Digital Network
ISP Internet Service Provider
IT Information Technology
ITSEC Information Technology SECurity
IVR Interactive Voice Response
KPI Key Performance Indicator
KQI Key Quality Indicator
MMS Multimedia Message Service
MMSC Multimedia Messaging Service Centre
MO Mobile Originate
MOS Mean Opinion Score
MT Mobile Terminate
OR Opinion Rating
PABX Private Automatic Branch eXchange
PDD Post Dialling Delay
PI Preliminary Information
PoP Point of Presence
POTS Plain Old Telephony Service
PSTN Public Switched Telephone Network
PTN Private Telecommunications Networks
QoS Quality of Service
RFC Request For Comment
SDS Short Data Service
SLA Service Level Agreement
SLO Service Level Objective
SMS Short Message Service
SP Service Provider
ST-MOS Listen Speech Quality Stability
TV Television
UE User Equipment
UMTS Universal Mobile Telecommunications System
WGR WAP Get Request
ETSI
11 ETSI EG 202 009-2 V1.3.1 (2014-12)
4 QoS methodology implementation
To implement the methodology detailed in clause 6 of ETSI EG 202 009-1 [i.11], the process consists first in defining
carefully the intended service, including its conditions of use and service level range, then to check for each step of the
customer relationship course and each QoS criterion what are the best suited metrics and indicators to express the user's
requirements and to monitor these indicators appropriately. When this is done for each cell of the ETSI
EG 202 009-1 [i.11] matrix, there will probably be too much indicators for a convenient handling, therefore only the
most relevant should be taken for publication or to include in a SLA.
Trying to limit drastically the number of indicators may be counter-productive as it gives the provider an incentive to
focus on a particular measure, perhaps at the expense of the general QoS. Also trade-offs may be necessary - for
example between cost and reliability. Benchmarking, when available, can provide useful support in this aspect. To
ensure the best compromise between the number of indicators and their ability to provide an effective assessment of the
QoS, it would be helpful to use surveys focusing on users' complaints to identify where are the main non quality issues
to adapt the indicator sample accordingly. This indicator selection should be reviewed regularly.
4.1 General principles for the metric and indicator definition
Every technical measurement should keep in with the customer perception (e.g. end-to-end transit time), although
suppliers may need to carry out technical measurements (KPI) on particular points to ensure a good customer perceived
quality even if these KPI are not directly perceptible to the user.
Some general principles should apply:
• Beware of mean values that gives figures which might be very far from a particular customer feeling.
• Focus on disturbance (should be 0) rather than performance (close to 100 %) even if it is only a presentation
issue.
• Use figures that may be aggregated (disturbance rate).
• For QoS targets define thresholds suited to the user perceived QoS.
When reading the present document, anyone should have in mind that a KQI is, in the user language, the translation of a
rate, a frequency or any other KPI, in a tangible perception from the user viewpoint. In this context, a KQI may involve
more than one KPI. The tables provided in the following clauses endeavour to give guidance on the relevant metrics and
indicators for the main electronic communication services and, when available, the corresponding standards.
Nevertheless, it should be clear that users can ask for different quality levels for the same service used in different
contexts. It is crucial to notice that QoS should always be assessed with regard to thresholds expected matching the
current user satisfaction in order to avoid a race to useless performances. Such QoS thresholds may be defined on a case
by case basis with, possibly, different requirements for different uses. In any case, the publication of KQI values is
expected to help to the users' freedom of choice, making them able to make an informed choice.
KQI for the service behaviour in the service utilization are in many cases service specific while in the other steps of the
customer relationship course they are often common to most services. Therefore to ease the reading, the definitions of
QoS metrics and indicators are split in two parts: those related to the operational aspects (Service utilization) and those
related to the other aspects of the service: sales, provision, alteration, upgrade, commercial and technical support,
complaint management, repair, charging/billing, network/service management, cessation.
It is important that the following points are made:
• What are the KQI that are pertinent to the particular service?
• How are these to be assessed?
• Who will assess them?
• What are the acceptable assessment procedures (test specification, i.e. ITU-T recommendation, ETSI standard
or survey and the frequency of measurement, sample size, confidence limits, etc.)?
• What are the acceptable, nominal service level ranges?
ETSI
12 ETSI EG 202 009-2 V1.3.1 (2014-12)
ETSI EG 202 009-1 [i.11] provides guidance on these aspects that have to be specified for any QoS assessment. In the
following clauses, tables are given to define the relevant QoS metrics and indicators for a selection of services along the
various aspects of the ICT customer relationship course.
Indicators may be assessed by various means: technical measurements performed by the supplier or an independent
organization, or a poll of a user panel.
According to users' opinion, in most cases both technical measurements and surveys among users are useful to achieve a
realistic assessment of the QoS.
Along with these considerations, the tables in the following clauses will consider various metrics for the QoS, seeking
to identify a set of measures that are expected to form the basis for judging the performance of the supplier from the
point of view of the consumers or business users. In this area the statistical quality is crucial to the credibility of the
results and should be given for most if not all of these measurements.
In these tables, the metric gives an expression of a criterion from the user viewpoint and relevant to his dashboard.
Indicators are used to give a quantifiable value allowing for the appraisal of the quality of a given service.
These tables are built on the basis of two general principles:
1) It is crucial, to avoid misunderstanding, that the metric and indicators to measure the QoS can be used and
managed by both the users and the providers, even if these metrics are viewed from different perspectives.
2) Theoretically, at least one KPI is necessary for an actual evaluation of each criterion and all the criteria are
needed for a comprehensive QoS appraisal of a given service aspect. The context for KQI is somewhat
different as a KQI may involve more than one criterion. The tables are built on the principle that all the criteria
are involved in at least one KQI. Nevertheless, as explained earlier, for practical reasons, a reduced set of
carefully selected indicators may be used for an efficient QoS monitoring.
Hence, each table shows, for each KQI, the metrics and the related indicator definitions. In addition, for each indicator,
the last column shows which type of measurement is appropriate.
As indicated earlier in this clause, the indicators given in these tables often refer to standards that provide additional
information about how the measurements should be performed and who is expected to perform and provide them. Any
one intending to assess QoS is invited to carefully read them. When there is no standard available to define appropriate
metrics and/or indicators, definitions are nevertheless proposed to compensate for the lack of such standard.
Since the previous version, a lot of efforts have been made to make available new standards for the assessment of KQI.
Not necessarily all the indicators defined in the standards are listed in clauses 5 and 6 tables but only those considered
as the most relevant to the users. In addition, it is to the organization using the present document to choose among these
indicators those that are appropriate to the particular situation to assess.
4.2 QoS requirements vs service element
Since nowadays, most services are in fact a service make-up, it is crucial before intending to assess the QoS of a service
(behaviour), to describe what the service is intended for, what are the conditions of use, what are the service elements
included, etc. In particular, for the GSM, UMTS, GPRS and ADSL services, the areas where the service is offered
should be specified.
Therefore, since the tables dealing with the indicators for the technical quality are service specific, the definition of the
functionality of the intended service is given at the beginning of the table.
In some cases, options can be bought to gain extended service features, e.g. flexible performances, high availability, etc.
These particular issues should be considered as service elements which can have their own QoS checking process and
might even impact the overall QoS. The QoS assessment of these specific service elements are tackled in the next
clauses.
4.2.1 Flexibility
Once the contract signed, it can be more or less easy to adjust some service settings according to the evolution of the
user needs without change in the main features of the service. This is why, according to the matrix provided in table 2
of clause 6.1 of ETSI EG 202 009-1 [i.11], such aspect of the flexibility is considered as a QoS requirement to be
assessed for each customer relationship step.
ETSI
13 ETSI EG 202 009-2 V1.3.1 (2014-12)
Nevertheless, in some other cases, the customer may wish to have the service features matching in real time the
requirements of particular type of use of the service. In such case, the flexibility can be considered as a service element
and therefore its QoS having to be analyzed with its own QoS checking process i.e. with regard to the criteria of
availability, integrity, time, capacity, etc.
Clause 7.2 is dedicated to the flexibility as service elements related to the service utilization, such as (re)configuration
and (re)provisioning.
4.2.2 Usability
Usability represents effectiveness, efficiency and satisfaction with which specified users can achieve specified goals
(tasks) in a particular environment with respect to the user profile.
As a first step the usability is considered as a QoS requirement to be assessed for each customer relationship step.
In a second time service elements such as transcoder, web browser could be defined.
4.2.3 Security
Security, data protection, privacy are key user concerns that have to be assessed for each customer relationship step. In
this context, according to ITSEC [i.35] IT security embraces multiple aspects that are detailed in clause 6 of Part 1 with
several service elements to achieve each of them.
Since, according to the matrix in table 2 of clause 6.1 of ETSI EG 202 009-1 [i.11], the present document is intended to
identify how the means available to assess the KPI can help to check that the provider contractual commitments are
met, all the security sections in clauses 5 and 6 are endeavouring to identify the KQI related to the key security
objectives defined according to the guidances provided in clause 6.1 of ETSI EG 202 009-1 [i.11]. In addition, if
needed, security can be assessed as a particular service element with regard to the QoS criteria and its impact on the
overall QoS of any step of the customer relationship course but particularly the operational ones. The clause 7.1 is
dedicated to the specific QoS criteria related to this security service element.
5 QoS metrics and indicators for the service behaviour
in the service utilization
It is very important to notice that the tables in this clause are intended to be focused on the services and to be
technology agnostic: being user oriented, the metric should be the same whatever the technology of the bearer network.
The differences due to differing technologies are expected to appear in the results, the presentation of them taking
possibly account of the communication path. These does not mean that a breakdown of the results according to the
technology or to the type of contract is irrelevant. Therefore, when the QoS assessment is expected to provide for a
comparison between the QoS achieved by several providers, a particular care should be given to the composition of the
test sample for each of them. Particularly when IP technology is used to provide the service, location, type of contract
and specification of the set-top-box of each sample item should be carefully selected to ensure the measurements
actually represent the QoS provided.
The services listed here are drawn from Recommendation ITU-T G.1010 [i.1] with some changes to take into account
the current market situation. The definition of the function achieved by each of them is given in the first sentence of the
related clause. Some services have usually several components and, when appropriate, indicators for each of such
components are identified separately in the list of the service indicators. Due to time and resource constraints, QoS
metrics and indicators are not available yet for all these services. As mentioned in clause 5.13, other services could be
provided in a later edition.
The methodology can be used to monitor the compliance of a provision to the provider commitments as well as a
performance assessment without any reference, for example in the intention to provide comparative information to a
prospect in order to help him to choose the offer best suited to his wishes.
It is worth noting that the reference to formal standards is crucial to the reliability of the results and to enable
comparisons between providers. On the other hand, the targets to meet are related to a contractual approach between the
provider and the regulator or between the provider and his customers.
ETSI
14 ETSI EG 202 009-2 V1.3.1 (2014-12)
What is at stake is that the user can know and possibly negotiate the targets on which the provider is committed and
then to get access to the information enabling him to check whether these commitments are met.
Users should be interested to have some guidance on which targets KQI should reach to ensure a good QoS but at this
stage of the work, it was not possible to provide such guidance. This will be for a later revision if the QoS measurement
experience has grown enough.
5.1 Audio broadcast - Audiostreaming
Audio broadcast/streaming: A mechanism whereby audio content can be rendered at the same time that it is being
transmitted to the client over the data network.
NOTE: The indicator evaluates the quality of audio stream.
In the context of end user evaluation without access to provider network, the assessment method should
perform a methodology "without reference".
Speech evaluation methodologies or speech quality assessment models are not appropriate to evaluate
audio quality. These methodologies or these models cannot be performed for audio quality evaluation.
In this context the most suitable approach is to determine "audio cut" (or "audio lack") occurrence and
low audio level sequence in the audio stream.
This analysis should be performed on the left and right channels, when available.
This indicator could be presented as the number of degradation or in ratio (number of degradation by time
unit).
An alternative assessment of the audio quality may be based on subjective tests or surveys by a
representative user panel (MOS value).
Availability
Metric Related indicator definition (KQI)
Rate of server accessibility Percentage of successful log-ins with respect to the total number of
attempts.
{Reference: Successful log-in ratio; ETSI EG 202 057-4 [i.15]}

Integrity
Metric Related indicator definition (KQI)
Listening audio loss ratio Consolidated duration of audible audio cut over one minute listening.
{Reference: Audio quality; ETSI ES 202 765-4 [i.20], clause 8.4}
Listening audio loss frequency Number of audible audio cut over one minute listening.
{Reference: Audio quality; ETSI ES 202 765-4 [i.20], clause 8.4}
Successful one minute listening ratio Number of continuous listening minutes without audio loss over the
required listening time.
{Reference: Audio quality; ETSI ES 202 765-4 [i.20], clause 8.4}

Time
Metric Rela
...