ETSI GS NFV-INF 010 V1.1.1 (2014-12)

Network Functions Virtualisation (NFV); Service Quality Metrics

DGS/NFV-INF010

General Information

• Status: Published
• Publication Date: 22-Dec-2014
• Technical Committee: NFV INF - Architecture of the Virtualisation Infrastructure
• Current Stage: 12 - Completion
• Due Date: 19-Jan-2015
• Completion Date: 23-Dec-2014

Standards Content (Sample)

ETSI GS NFV-INF 010 V1.1.1 (2014-12)






GROUP SPECIFICATION
Network Functions Virtualisation (NFV);
Service Quality Metrics
Disclaimer
This document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry Specification
Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

---------------------- Page: 1 ----------------------
2 ETSI GS NFV-INF 010 V1.1.1 (2014-12)



Reference
DGS/NFV-INF010
Keywords
measurement, NFV, quality
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2014.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI GS NFV-INF 010 V1.1.1 (2014-12)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
Introduction . 4
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions and abbreviations . 7
3.1 Definitions . 7
3.2 Abbreviations . 8
4 NFV Service Quality Metrics Taxonomy. 9
5 Virtual Machine Service Quality Metrics. 11
5.1 VM Stall . 13
5.2 Premature VM Release Ratio . 13
5.3 VM Scheduling Latency . 14
5.4 VM Clock Error . 14
6 Virtual Network Interface Service Quality Metrics . 15
7 Technology Component Service Quality Metrics . 16
8 Orchestration Service Quality Metrics . 17
9 Service Quality Metrics Use Case . 18
9.1 Short Term Use of Service Quality Metrics . 18
9.2 Medium Term Use of Service Quality Metrics . 19
9.3 Long Term Use of Service Quality Metrics . 19
10 Recommendations . 19
10.1 Measurement of Service Quality Metrics . 19
10.2 Service Quality Metrics in SLAs . 20
10.3 Detailed Metric Definitions . 20
10.4 Methods of Measurement . 20
10.5 Reporting Statistics and Results Processing . 20
10.6 Characterization Plans . 20
Annex A (informative): Example NFVI-VIM Interactions related to Service Quality Metrics . 21
A.1 Introduction . 21
A.2 Resource Establishment (VM) . 21
A.2.1 VM Provisioning Latency . 21
A.2.2 VM Provisioning Reliability . 22
A.2.3 VM DOA Ratio . 22
A.2.4 VM Placement Policy Compliance . 23
A.3 Resource Establishment (Infrastructure Network) . 23
A.3.1 VN Provisioning Latency . 24
A.3.2 VN Provisioning Reliability . 24
Annex B (informative): Authors & contributors . 25
Annex C (informative): Bibliography . 26
History . 27

ETSI

---------------------- Page: 3 ----------------------
4 ETSI GS NFV-INF 010 V1.1.1 (2014-12)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Group Specification (GS) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
The present document deals with specific aspects of Service Quality Metrics in the context of Network Function
Virtualisation.
Infrastructure Architecture Document Document #
Overview GS NFV INF 001
Illustrative Use Cases for the NFV Infrastructure GS NFV INF 002
Architecture of the Infrastructure Compute Domain GS NFV INF 003
Domains Hypervisor Domain GS NFV INF 004
Infrastructure Network Domain GS NFV INF 005
Architectural Methodology Interfaces and Abstraction GS NFV INF 007
Service Quality Metrics GS NFV INF 010

Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "may not", "need", "need not", "will",
"will not", "can" and "cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms
for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
As shown in figure 1, the service quality delivered by a VNF instance to end users is dependent on the service quality of
the compute, network and other resources delivered by NFV infrastructure, VIM, VNFM and NFVO to the VNF
instance. Objective and quantitative metrics for the service delivered by NFV infrastructure and orchestration to NFV
consumers enables:
• Better engineering of VNF user service quality.
• Efficient fault localization and mitigation.
• Faster identification of true root cause of service impairment so proper corrective actions can be taken
promptly.
ETSI

---------------------- Page: 4 ----------------------
5 ETSI GS NFV-INF 010 V1.1.1 (2014-12)
Quantitative, objective metrics
for service quality delivered by
Assuring that VNF service
NFV cloud service provider to NFV
quality delivered to users is
acceptable… consumers enables:
better engineering of
application (VNF) user service
quality,
efficient fault localization and
mitigation, and
Faster identification of true
root cause of service impairments
…relies on acceptable NFV
so proper corrective actions can
infrastructure service being
be taken promptly
delivered to VNF components

Figure 1: Purpose of NFV Service Quality Metrics
Figure 2 illustrates the service boundary characterized by these metrics between canonical NFV consumers who operate
VNFs and NFVI service providers who operate NFV infrastructure and supporting systems. Objective metrics of the
service quality delivered by the NFV infrastructure service provider to the NFV consumer's VNFs enable quantitative
discussions and agreements for the objectives of NFV service quality to assure that the NFV consumer's VNF instances
deliver acceptable service quality to end users.
NFV Consumer NFVI (IaaS)
Service Provider
End Users
NFV service quality metrics
enable VNF suppliers, NFV
consumers and NFV IaaS
service providers to agree on
cloud service quality
objectives that enable VNFs to

deliver acceptable service to
the NFV consumer’s end users

Figure 2: Service Quality Metrics in the Context of NFV Reference Architecture
Both new and familiar metrics will be needed to complete the quantification of service quality. The present document
examines the properties of the virtual infrastructure and describes metrics relevant and useful to virtualised functions.
ETSI

---------------------- Page: 5 ----------------------
6 ETSI GS NFV-INF 010 V1.1.1 (2014-12)
1 Scope
The present document enumerates metrics for NFV infrastructure, management and orchestration service qualities that
can impact the end user service qualities delivered by VNF instances hosted on NFV infrastructure. These service
quality metrics cover both direct service impairments, such as IP packets lost by NFV virtual networking which impacts
end user service latency or quality of experience, and indirect service quality risks, such as NFV management and
orchestration failing to continuously and rigorously enforce all anti-affinity rules which increases the risk of an
infrastructure failure causing unacceptable VNF user service impact. Performance relationships exist between the
metrics described in this document and in other specifications such as [i.5].
The present document does not consider:
1) Units of measurement for reporting, such as whether VM premature release rates should be expressed as
hourly rate (e.g. 0,0001 premature VM release events per hour), annualized rate (e.g. 0,88 premature VM
release events per year), hours between events (e.g. 10 000 hour mean time between premature release events),
or events per other unit of time (e.g. 100 000 FITs, meaning 100 000 premature release events in one billion
hours of operation).
2) Methods of Measurement which stipulate exactly how metrics will be measured.
3) Rigorous counting and exclusion rules, like the precise details given in the TL 9000 Measurements Handbook
[i.13].
4) Metrics that do not directly or indirectly impact VNF user service quality, like power efficiency.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI GS NFV-INF 001: "Network Functions Virtualisation (NFV); Infrastructure Overview".
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] NIST Special Publication 800-145 (September 2011): "The NIST Definitions of Cloud
Computing", Peter Mell and Timothy Grance, US National Institute of Standards and Technology.
[i.2] IETF RFC 2330: "Framework for IP Performance Metrics".
ETSI

---------------------- Page: 6 ----------------------
7 ETSI GS NFV-INF 010 V1.1.1 (2014-12)
[i.3] Wiley-IEEE Press, 2013: "Service Quality of Cloud-Based Applications," Eric Bauer and Randee
Adams.
[i.4] ETSI GS NFV-MAN 01: "Network Functions Virtualisation (NFV); Management and
Orchestration".
[i.5] draft-ietf-ippm-model-based-metrics-02 (work in progress) (February 2014): "Model Based Bulk
Performance Metrics", M. Mathis and A. Morton.
[i.6] ETSI GS NFV-PER 001: "Network Functions Virtualisation (NFV); NFV Performance &
Portability Best Practises".
[i.7] IETF RFC 6390: "Guidelines for Considering New Performance Metric Development".
[i.8] ISO/IEC 15939:2007: "Systems and software engine-ing -- Measurement process".
[i.9] NIST draft Cloud Service Metric Description v2.
[i.10] Recommendation ITU-T M.3341: "Requirements for QoS/SLA management over the TMN X-
interface for IP-based services".
[i.11] Recommendation ITU-T Y.1543: "Measurements in IP networks for inter-domain performance
assessment".
[i.12] Recommendation ITU-T I.356: "B-ISDN ATM layer cell transfer performance".
[i.13] TL 9000 Measurements Handbook, release 5.0, July 2012, QuestForum.
NOTE: Available at http://www.tl9000.org/handbooks/measurements_handbook.html.
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
derived metric: metric defined on the basis of the values produced by other Metrics
NOTE: An example from packet transfer performance is delay variation, which is based on multiple values
produced from measurement of a packet delay metric. (This definition is consistent with
IETF RFC 2330 [i.2] and the derived performance parameter in Recommendation ITU-T I.356 [i.12]).
measurement: set of operations having the object of determining a Measured Value or Measurement Result
NOTE: The actual instance or execution of operations leading to a Measured Value. (Based on the definition of
Measurement in IETF RFC 6390 [i.7], as cited in ISO/IEC 15939 [i.8]).
measurement point: physical or logical point at which observations are made and to which the measure obtained is
related, e.g. a boundary or point of demarcation between domains or functional entities (derived from NIST draft Cloud
Service Metric Description v2 [i.9])
NOTE 1: When one or more measurement points are specified along with a performance metric, they define the
scope of measurement and should be included with the measurement results for accurate interpretation.
NOTE 2: A point in a system possessing sufficient functionality to provide observations of reference events.
(derived from Recommendation ITU-T M.3341 [i.10]) The interface between two communicating entities
is often designated as a Measurement Point.
ETSI

---------------------- Page: 7 ----------------------
8 ETSI GS NFV-INF 010 V1.1.1 (2014-12)
metric: standard definition of a quantity, produced in an assessment of performance and/or reliability of the network,
which has an intended utility and is carefully specified to convey the exact meaning of a measured value
NOTE: This definition is consistent with that of Performance Metric in IETF RFC 2330 [i.2] and
ETSI GS NFV-PER 001 [i.6].
EXAMPLE: Packet transfer performance or reliability of a network.
parameter: input factor defined as a variable in the definition of a Metric
NOTE: A numerical or other specified factor forming one of a set that fully-defines a Metric or sets the
conditions of its operation. Most Parameters do not change the fundamental nature of the metric's
definition, but others have substantial influence. All Parameters should be known in order to conduct
measurements conforming to a Metric and interpret the results. An example Parameter includes the
Measurement Point(s). (derived from IETF work in progress to design a Registry for Performance
Metrics, consistent with IETF IPPM Literature)
reference event: transfer of a discrete unit of control or user information encoded in accordance with a specific
protocol across a Measurement Point
NOTE: Complementary classes of reference events can sometimes be distinguished: exit and entry events, or start
and stop events (derived from Recommendation ITU-T Y.1543 [i.11]).
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
CCDF Complementary Cumulative Distribution Function
CPU Central Processing Unit
DOA Dead on Arrival
NOTE: Also referred to as "Out-of-Box" (OOB) failures.
HW&SW Hardware and Software
IETF Internet Engineering Task Force
IP Internet Protocol
MIB Management Information Base
NFV Network Function Virtualisation
NFVI NEFV Infrastructure
NFVO NFV Orchestrator
NIC Network Interface Card
OS Operating System
SLA Service Level Agreement
SLO Service Level Objective
SQM Service Quality Metrics
TcaaS Technology Component offered as-a-Service
NOTE: Like Database-as-a-Service.
UTC Universal Coordinated Time
VIM Virtual Infrastrucutre Management
VM Virtual Machine
VN Virtual Network
VNF Virtualised Network Function
VNFC Virtualised Network Function Component
VNFM VNF Manager
ETSI

---------------------- Page: 8 ----------------------
9 ETSI GS NFV-INF 010 V1.1.1 (2014-12)
4 NFV Service Quality Metrics Taxonomy
End users experience services delivered by VNF instances, which are implemented via suites of VNFCs working
together. The services delivered to end users by individual VNFC instances is dependent on the service quality of the
virtual machine instance that hosts the component and the virtual network service that delivers connectivity to other
VNFCs. End user service quality of VNFs which use functional blocks or technology components that are offered
'as-a-Service' like Database-as-a-Service or Load-Balancing-as-a-Service are also vulnerable to service quality
impairments of those technology components. NFV management and orchestration can also presents subtle risks to
VNF service quality if elastic resource growth or repair is slow or faulty, or if the VNF's anti-affinity rules are not
strictly and continuously enforced. Figure 3 visualizes the four suites of NFV service quality metrics:
1) Virtual machine service quality metrics.
2) Virtual network service quality metrics.
3) Technology components offered 'as-a-Service' (e.g. Database-as-a-Service) quality metrics.
4) Orchestration service quality metrics.
VNF (virtualized network function)
is a cloud-based application
Virtual Machine KQI
service boundary
Virtual Network KQI
service boundary
Orchestration KQI
service boundary
NFV
Infrastructure
KQIs for technology
components offered as-a-
Service (e.g., Database-as-
a-Service) are not shown

Figure 3: NFV Service Quality Metrics on High Level Overview of NFVI
When assessing performance of the NFV Infrastructure, measurement points at the [Nf-Vi]/* reference points will be
critical. Measurement points at other locations in the NFV architecture will necessarily include the performance
contribution of management components, such as the Vi-Vnfm and Or-Vi reference points, and these may also prove to
be valuable aids in performance management.
The NFV service quality metrics are summarized in table 1. Each cell of the matrix is associated with a Service/life-
cycle category (e.g. Orchestration, Operation) and a quality criterion (Speed, Accuracy, Reliability). Some intersections
are critical for certain functions or resources while others may be inapplicable or contain secondary metrics. By listing
each Quality Metric in its corresponding cell, it is possible to identify overlaps and gaps. The matrix may also facilitate
the process to determine which Quality Metrics are the key ones and should be measured, collected, and reported. The
Service Quality Metrics are applicable when the entity measured is deemed to be in the Available state (through
continuous evaluation of one or more metrics). See [i.3] for consistent definitions of Availability and Reliability.
ETSI

---------------------- Page: 9 ----------------------
10 ETSI GS NFV-INF 010 V1.1.1 (2014-12)
Table 1: Summary of NFV Service Quality Metrics
Service Metric Category Speed Accuracy Reliability
Orchestration Step 1 VM Provisioning Latency VM Placement Policy VM Provisioning Reliability
(e.g. Resource Allocation, Compliance VM Dead-on-Arrival (DOA)
Configuration and Setup) Ratio
VirtualMachine operation VM Stall (event duration and VM Clock Error VM Premature Release
frequency) Ratio
VM Scheduling Latency
Virtual Network VN Provisioning Latency VN Diversity Compliance VN Provisioning Reliability
Establishment
Virtual Network operation Packet Delay Packet Loss Ratio Network Outage
Packet Delay Variation
(Jitter)
Delivered Throughput
Orchestration Step 2  Failed VM Release Ratio
(e.g. Resource Release)
Technology Component as- TcaaS Service Latency - TcaaS Reliability
a-Service (e.g.defective transaction
ratio)
TcaaS Outage

The impact of each NFV service quality metric of table 1 can directly or indirectly impair the VNF user service quality
as follows:
• VM provisioning latency and VM provisioning reliability directly impact the time it takes to elastically
grow online VNF service capacity or to restore full VNF redundancy (i.e. eliminate simplex exposure)
following a failure event.
• VM Dead on Arrival (DOA) indirectly impacts the time to elastically grow or repair VNF capacity because a
latent VM fault shall be detected and somehow mitigated before needed VNF capacity can enter user service.
• VM premature release ratio directly impacts the frequency that VNF service recovery actions (e.g. high
availability failovers) shall be taken.
• VM stall characterizes disruptions in prompt and continuous execution of VNFC software which impacts the
service latency and quality of service enjoyed by end users.
• VM scheduling latency characterizes how promptly VNFC software is executed, such as when processing
isochronous bearer plane traffic, which impacts the service latency and quality of service enjoyed by end users.
• VM clock error characterizes the realtime inaccuracy presented to VNFC software for billing, fault and other
records that rely on accurate timestamping.
• VM placement policy compliance characterizes how reliably the NFV infrastructure provider continuously
and correctly enforces the VNF's anti-affinity rules, thereby minimizing the risk that a single infrastructure
failure event will produce an unacceptable VNF user service quality impact.
• Packet delay characterizes the incremental user service latency introduced by communications between a
VNF's VNFCs, which impacts the service latency and quality of service enjoyed by end users. A key input
parameter for any packet transfer metric is the offered load during the measurement.
• Packet delay variation (jitter) is a derived metric that characterizes the incremental user service delay
variation introduced by instability in communications latency between VNFCs within a VNF, which impacts
the service latency and quality of service enjoyed by end users.
• Delivered Throughput is a derived metric from the offered load input parameter and other packet transfer
performance metrics (loss, delay) measured at that load to characterize the actual capacity of communications
between a VNF's VNFCs, and which impacts the quality of service enjoyed by end users
(ETSI GS NFV-MAN 001 [i.4]).
ETSI

---------------------- Page: 10 ----------------------
11 ETSI GS NFV-INF 010 V1.1.1 (2014-12)
• Packet loss ratio impacts end user service latency, reliability and quality because lost packets shall be
detected, and mitigated via retry, retransmission or concealment, which impacts the service latency and quality
of service enjoyed by end users. Lost packets should be assessed at measurement points near the ingress and
egress of the network being measured. Again, a key input parameter for any packet transfer metric is the
offered load during the measurement.
• Network outage loss of virtual network connectivity directly impacts the service latency, quality and
availability experienced by end users. Network impairment episodes that persist longer than the VNF's
Maximum Acceptable Network Transient Time parameter will prompt highly available VNFs to automatically
initiate service recovery actions, up to and including VNF disaster recovery actions.
• Failed VM Release Ratio characterizes how reliably the usage records for VM release are recorded.
• VN provisioning latency and VN provisioning
...