ETSI GS MEC-IEG 006 V1.1.1 (2017-01)

Mobile Edge Computing; Market Acceleration; MEC Metrics Best Practice and Guidelines

DGS/MEC-IEG006Metrics

General Information

• Status: Published
• Publication Date: 30-Jan-2017
• Technical Committee: MEC IEG - Industry Enabling Group
• Current Stage: 12 - Completion
• Due Date: 31-Jan-2017
• Completion Date: 31-Jan-2017

Standards Content (Sample)

ETSI GS MEC-IEG 006 V1.1.1 (2017-01)






GROUP SPECIFICATION
Mobile Edge Computing;
Market Acceleration;
MEC Metrics Best Practice and Guidelines
Disclaimer
The present document has been produced and approved by the Mobile Edge Computing (MEC) ETSI Industry Specification
Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

---------------------- Page: 1 ----------------------
2 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)



Reference
DGS/MEC-IEG006Metrics
Keywords
MEC, KPI

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2017.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions and abbreviations . 7
3.1 Definitions . 7
3.2 Abbreviations . 8
4 Metrics . 9
4.1 General . 9
4.2 Latency . 10
4.2.1 General . 10
4.2.2 Round-Trip Time . 10
4.2.3 One-Way Delay (OWD) . 11
4.2.4 Set-up Time . 11
4.2.5 Service Processing Time . 11
4.2.6 Context-update time . 12
4.3 Energy efficiency . 12
4.4 Network throughput . 13
4.5 System resource footprint . 14
4.5.1 General . 14
4.5.2 Computational load . 14
4.5.3 Non user data volume exchange . 14
4.6 Quality . 14
4.6.1 General . 14
4.6.2 Objective and service-independent metrics about quality. 14
4.6.3 Objective and service-dependent metrics about quality . 15
4.6.4 Subjective and service-dependent metrics about quality . 15
4.6.5 Objective metrics about user comfort . 15
5 Measurement methodology . 16
5.1 General . 16
5.2 Evaluation of latency . 16
5.2.0 Introduction. 16
5.2.1 Measurement methodology . 17
5.2.1.1 Peak workload test . 17
5.2.1.2 Uniform workload tests . 17
5.2.1.3 Stress tests . 17
5.2.2 Latency measurement setup 1: passive measurements at the terminal . 17
5.2.3 Latency measurement setup 2: passive measurements by probes . 18
5.2.4 Latency measurement setup 3: active measurements . 19
5.3 Evaluation of energy efficiency . 19
5.3.1 Introduction. 19
5.3.2 Measurement methodology . 20
5.3.2.1 General . 20
5.3.2.2 Energy efficiency measurement setup 1 (network side) . 20
5.3.2.2.1 General considerations . 20
5.3.2.2.2 Baseline: measurement without the MEC Server . 21
5.3.2.2.3 Frontline: measurement with the MEC Server . 21
5.3.2.2.4 Computation of EE gains . 21
5.3.2.3 Energy efficiency measurement setup 2 (terminal side) . 22
ETSI

---------------------- Page: 3 ----------------------
4 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)
5.3.2.3.1 General considerations . 22
5.3.2.3.2 Baseline: measurement without the MEC Server . 22
5.3.2.3.3 Frontline: measurement with the MEC Server . 22
5.3.2.3.4 Computation of EE gains . 23
5.4 Evaluation of network throughput . 23
5.4.1 General . 23
5.4.2 Network throughput measurement setups . 23
5.5 Evaluation of resource footprint . 24
5.5.1 General . 24
5.5.2 Computational load measurement setup 1: isolated execution environment . 24
Annex A (informative): Network Throughput Example . 25
Annex B (informative): Examples of metric value ranges . 26
B.1 5G latency requirements . 26
B.2 5G energy efficiency . 26
Annex C (informative): POC#3 RAVEN - example of latency metric assessment . 27
History . 29

ETSI

---------------------- Page: 4 ----------------------
5 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Group Specification (GS) has been produced by ETSI Industry Specification Group (ISG) Mobile Edge
Computing (MEC).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
Mobile Edge Computing is a new technology that provides an IT service environment and cloud-computing capabilities
at the edge of the mobile network, in close proximity to mobile subscribers. In order to make MEC a success and
encourage network operators to deploy Mobile Edge (ME) systems as well as to make MEC attractive to application
developers and service providers, it is necessary to demonstrate the benefits of this technology for fulfilling various
requirements. In order to make MEC an attractive proposition for service providers and applications developers to host
their applications on a ME Host instead of in a centralized cloud, it is important to demonstrate a quantifiable
performance increase.
The present document describes a number of performance metrics which can be used to demonstrate the benefits of
deploying services and applications on a ME Host compared to a centralized cloud or server. Examples of how these
metrics can be measured are also described.
Examples of such metrics KPIs are reducing latency, increasing end-to-end energy efficiency and increasing network
throughput.
ETSI

---------------------- Page: 5 ----------------------
6 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)
1 Scope
The present document describes various metrics which can potentially be improved through deploying a service on a
MEC platform. Example use cases are used to demonstrate where improvements to a number of key performance
indicators can be identified in order to highlight the benefits of deploying MEC for various services and applications.
Furthermore, the present document describes best practices for measuring such performance metrics and these
techniques are further exemplified with use cases.
Metrics described in the present document can be taken from service requirements defined by various organizations
rd
(e.g. 5G service requirements defined by Next Generation Mobile Networks (NGMN) or 3 Generation Partnership
Project (3GPP)). An informative annex is used to document such desired and/or achieved ranges of performance which
could be referenced from the main body of the present document.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI ES 202 706 (V1.4.1): "Environmental Engineering (EE); Measurement method for power
consumption and energy efficiency of wireless access network equipment".
[2] ETSI ES 203 228 (V1.1.1): "Environmental Engineering (EE); Assessment of mobile network
energy efficiency".
[3] ETSI GS MEC 002: "Mobile Edge Computing (MEC); Technical Requirements".
[4] ETSI GS MEC 001: "Mobile Edge Computing (MEC); Terminology".
[5] ETSI ES 202 336-12: "Environmental Engineering (EE); Monitoring and control interface for
infrastructure equipment (power, cooling and building environment systems used in
telecommunication networks); Part 12: ICT equipment power, energy and environmental
parameters monitoring information model".
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] IETF RFC 4656: "One way active measurement protocol".
[i.2] IETF RFC 5357: "A two-way active measurement protocol".
ETSI

---------------------- Page: 6 ----------------------
7 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)
[i.3] IETF IP Performance Metrics Working Group: IPPM status pages.
NOTE: Available at https://tools.ietf.org/wg/ippm/.
[i.4] IETF IP Performance Metrics Working Group: Charter.
NOTE: Available at https://tools.ietf.org/wg/ippm/charters.
[i.5] NGMN Alliance 5G White Paper version 1.0 (17 February 2015): "NGMN 5G White Paper".
NOTE: Available at https://www.ngmn.org/uploads/media/NGMN_5G_White_Paper_V1_0.pdf.
[i.6] J. S. Milton, J. Arnold, "Introduction to Probability and Statistics", McGraw-Hill Education,
th
4 Edition.
[i.7] P. Serrano, M. Zink, J. Kurose, "Assessing the fidelity of COTS 802.11 sniffers", IEEE
INFOCOM 2009, Rio de Janeiro, Brazil, April 2009.
[i.8] P. Serrano, A. Garcia-Saavedra, G. Bianchi, A. Banchs, A. Azcorra, "Per-frame Energy
Consumption in 802.11 Devices and its Implication on Modeling and Design," IEEE/ACM
Transactions on Networking, vol.23, no.4, pp.1243-1256, Aug. 2015.
[i.9] N Vallina-Rodriguez, J Crowcroft, "Energy Management Techniques in Modern Mobile
Handsets," IEEE Communications Surveys & Tutorials, 1-20.
[i.10] ETSI MEC PoC#3 RAVEN: "Radio aware video optimization in a fully virtualized network".
NOTE: Available at
http://mecwiki.etsi.org/index.php?title=PoC_3_Radio_aware_video_optimization_in_a_fully_virtualized
_network.
[i.11] ETSI GS MEC 015: "Mobile Edge Computing (MEC) Bandwidth Management API".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the terms and definitions given in ETSI GS MEC 001 [4], ETSI
ES 203 228 [2] and the following apply:
NOTE: For some background definitions for network level energy efficiency, see ETSI ES 203 228 [2].
Energy Efficiency (EE): relation between the useful output and energy/power consumption
mobile network coverage Energy Efficiency: ratio between the area covered by the network in the Mobile Network
under investigation and the energy consumption
mobile network data Energy Efficiency: ratio between the performance indicator based on Data Volume and the
energy consumption when assessed during the same time frame
mobile network energy consumption: overall energy consumption of equipment included in the MN under
investigation
system resources: any kinds of entities to be shared to compose services including computing power, processor and
accelerator loads, memory usage, storage, network, database and applications
NOTE: System resources can be considered as a set of coherent functions, network data objects or services,
accessible through a server where such system resources reside on a single host or multiple hosts and are
clearly identifiable.
ETSI

---------------------- Page: 7 ----------------------
8 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
rd
3GPP 3 Generation Partnership Project
API Application Programming Interface
BER Bit Error Rate
CN Core Network
CPU Central Processing Unit
DC Direct Current
EE Energy Efficiency
eNB eNodeB
GPS Global Positioning System
ICMP Internet Control Message Protocol
IDT Inter Departure Time
IP Internet Protocol
IPPM IP Performance Metrics
KPI Key Performance Indicator
ME Mobile Equipment
MN Mobile Network
MOS Mean Opinion Score
MSL MEC-Specific Latency
MSS Maximum Segment Size
MTU Maximum Transmission Unit
NGMN Next Generation Mobile Networks
NRQA No Reference Quality Assessment
NRT Non Real-Time
NTP Network Time Protocol
OS Operating System
OWD One-Way Delay
PA Power Amplifier
PEAQ Perceptual Evaluation of Audio Quality
PEVQ Perceptual Evaluation of Video Quality
PLR Packet Loss Rate
POC Proof Of Concept
POLQA Perceptual Objective Listening Quality Assessment
PSNR Peak Signal-to-Noise Ratio
PSS Proportional Set Size
PTP Precision Time Protocol
QoS Quality of Service
RAN Radio Access Network
RAVEN Radio Aware Video optimization in a fully virtualized network
RSS Resident Set Size
RT Real-Time
RTT Round-Trip Time
SDT Service Delivery Time
SGW Service GW
SPT Service Processing Time
SUT Set-Up Time
TCP Transmission Control Protocol
UD Update delay
UDP User Datagram Protocol
UE User Equipment
USS Unique Set Size
VSS Virtual Set Size
ETSI

---------------------- Page: 8 ----------------------
9 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)
4 Metrics
4.1 General
This clause introduces the metrics considered by ETSI ISG MEC for the evaluation of improvements introduced by
Mobile Edge Computing technologies. While clause 4 is describing all the different metrics considered (in separated
clauses), clause 5 is organized similarly (with one clause corresponding to each metric in clause 4) in order to introduce
the related measurement methodologies.
Generally MEC metrics are introduced with different purposes: evaluating the improvement given by MEC (as
perceived by the end user), and assessing the benefits of different MEC deployment options (thus giving insights from a
technologic point of view).
All metrics introduced in the present document can demonstrate the improvements of MEC solutions at least in the two
following ways:
1) comparison between MEC and non-MEC solutions;
2) assessment of MEC deployments: comparison between different ME host positions within the network.
In both cases, the goal is not to compare different vendors or solution providers, but to assess the improvement of MEC
introduction with respect to a traditional system (without MEC), e.g. in order to understand the different deployment
options against the different use cases (e.g. by minimizing costs, maximizing benefits or flexibility).
For this reason, MEC metrics can be classified into two main groups: functional and non-functional metrics. For both
categories (defined here below), metrics can be referred to different MEC use cases, as listed in IETF RFC 4656 [i.1],
and the actual assessment of these metrics can depend on the particular service and/or application utilization:
1) Functional metrics are related to MEC performances impacting on user perception (often called also KPIs, key
performances indicators):
- Examples of functional service performance KPIs include: latency (both end-to-end, and one-way),
energy efficiency, throughput, goodput, loss rate (number of dropped packets), jitter, number of
out-of-order delivery packets, QoS, and MOS. Each of the functional metrics should be defined on per
service basis. Note that the latency in localization (time to fix the position) is different from latency in
content delivery.
2) Non-functional metrics are related to the performance of the service in terms of deployment and management:
- Examples of non-functional metrics include: service lifecycle (instantiation, service deployment, service
provisioning, service update (e.g. service scalability and elasticity), service disposal), service availability
and fault tolerance (aka reliability), service processing/computational load, global ME host load, number
of API request (more generally number of events) processed/second on ME host, delay to process API
request (north and south), number of failed API request. The sum of service instantiation, service
deployment, and service provisioning provide service boot-time.
In both cases, one could measure all the statistics over the above metrics. In fact, all metrics are in principle
time-variable, and could be measured in a defined time interval and described by a profile over time or summarized
through:
• the maximum value;
• mean and minimum value;
• standard deviation;
• the value of a given percentile;
• etc.
All MEC metrics assessments can be done by considering the overall system, or portions of that, according to the
purpose of the measurement itself. An example below (figure 1) shows a mobile network system with ME host, and the
different entities potentially involved in the assessment.
ETSI

---------------------- Page: 9 ----------------------
10 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)

Figure 1: Measuring MEC metrics
4.2 Latency
4.2.1 General
The concept of latency is wide and encompasses manifolds metrics: in communications, latency refers to a time-interval
whose measurement quantifies the delay elapsed between any event and a consequent target effect. Even more, still in
the communication domain, latency is useful to measure phenomena both in the control plane (e.g. set-up time or
hand-over time) and in the data plane (e.g. transfer delay). The purpose of this clause is not to define all the latency
metrics potentially relevant to the MEC solutions, but rather to highlight what type of latency metrics can be adopted
(or newly defined) and their potential roles.
Referring to all the latency metrics in the clauses 4.2.2 to 4.2.6, it is assumed that an ideal synchronization holds across
the nodes under test for measurements purposes.
Note that different Latency measurements have been specified in IETF RFC 4656 [i.1] and IETF RFC 5357 [i.2].
However, the latency definitions within the subsequent clauses are referring to latency measured on application level.
4.2.2 Round-Trip Time
Round-Trip Time (RTT): by referring to figure 1, it is defined as the time taken for a request (e.g. packet) generated
from a terminal (a) to go to the destination, be updated or replied and travel back to (a), in conditions of ideal service
capabilities (i.e. the server and/or terminal response time is supposed to be fixed and the RTT does not depend on the
server/terminal computational load). Characteristics of RTT include:
1) Depending on the service type, the RTT might include very heterogeneous paths. Referring to figure 1:
- (a)-(c)-(a) in case of MEC client-server applications;
- (a)-(e)-(a) in case of non-MEC client-server applications;
ETSI

---------------------- Page: 10 ----------------------
11 ETSI GS MEC-IEG 006 V1.1.1 (2017-01)
- (a)-(c)-(a') -(c)-(a) in case of MEC P2P applications;
- (a)-(e)-(a') -(e)-(a) in case of non-MEC P2P applications.
2) RTT is a variable which likely changes over time for the same station and is described by a RTT profile over
time. RTT statistics might be summarized through: the maximum, mean and minimum value of RTT, the
variance, the value of a given percentile, etc.
3) RTT might also vary throughout different stations (e.g. terminal (a) and terminal (a')). A statisti
...