ETSI GS NFV-PER 001 V1.1.1 (2014-06)

Network Functions Virtualisation (NFV); NFV Performance & Portability Best Practises

DGS/NFV-PER001

General Information

• Status: Published
• Publication Date: 29-Jun-2014
• Technical Committee: NFV PER - Performance & Portability
• Current Stage: 12 - Completion
• Due Date: 15-Jul-2014
• Completion Date: 30-Jun-2014

Standards Content (Sample)

ETSI GS NFV-PER 001 V1.1.1 (2014-06)






Group Specification
Network Functions Virtualisation (NFV);
NFV Performance & Portability Best Practises
Disclaimer
This document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry Specification
Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

---------------------- Page: 1 ----------------------
2 ETSI GS NFV-PER 001 V1.1.1 (2014-06)



Reference
DGS/NFV-PER001
Keywords
performance, portability
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2014.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI GS NFV-PER 001 V1.1.1 (2014-06)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions and abbreviations . 7
3.1 Definitions . 7
3.2 Abbreviations . 8
4 Introduction . 10
5 Methodology and relation to NFV use cases . 14
6 Bottleneck analysis and relevant features for high and predictable performance . 18
6.1 Data plane workload in an intermediate element . 18
6.1.1 High and predictable performance on bare metal . 18
6.1.2 High and predictable performance on a virtualised environment . 20
7 Best practises and recommendations . 23
7.1 HW architecture and capabilities . 23
7.2 Hypervisor capabilities . 23
8 Templates for portability . 24
8.1 Compute Host Descriptor (CHD) . 24
8.1.1 List of capabilities . 24
8.1.1.1 Processor capabilities . 24
8.1.1.2 Memory capabilities . 25
8.1.1.3 Hypervisor capabilities . 26
8.1.1.4 Resource topology and availability . 26
8.2 Virtual Machine Descriptor (VM Descriptor) . 29
8.2.1 Context on VM Image requirements . 29
8.2.2 List of requirements . 29
8.2.2.1 Processor requirements . 30
8.2.2.2 Memory requirements . 30
8.2.2.3 Hypervisor requirements . 30
8.2.2.4 Number of resources and topology . 31
8.2.2.5 Impact from/to other VMs . 33
Annex A (informative): Gap analysis of hypervisor and cloud OS . 35
Annex B (informative): Relevant technologies . 36
B.1 HW support for Virtualisation . 36
B.2 Direct I/O access to processor cache . 36
B.3 Single Root I/O Virtualisation (SR-IOV) . 37
B.4 Remote Direct Memory Access (RDMA) . 37
B.5 Infiniband . 37
Annex C (informative): NFV Test Methodologies . 38
C.1 Control Plane Testing . 38
C.1.1 Common Testing Methodologies for physical world and virtual world . 38
ETSI

---------------------- Page: 3 ----------------------
4 ETSI GS NFV-PER 001 V1.1.1 (2014-06)
C.1.2 NFV specific control plane testing . 38
C.1.2.1 Scalability . 39
C.1.2.1.1 Test Setup . 39
C.1.2.1.2 Test Procedure . 39
C.1.2.1.3 Desired Results . 41
C.1.2.2 Failover Convergence Testing . 41
C.1.2.2.1 Test Setup . 41
C.1.2.2.2 Test Procedure . 42
C.1.2.3 VM Migration . 43
C.1.2.3.1 Test Setup . 43
C.1.2.3.2 Test Procedure . 44
C.1.2.3.3 Result Analysis . 44
C.2 Data Plane Testing . 45
C.2.1 NFV specific Data Plane Testing . 45
C.2.1.1 Example of performance benchmarking of a DPI device in different configurations . 45
C.2.1.1.1 Test Setup . 45
C.2.1.1.2 Test Procedure . 46
C.2.1.1.3 Desired Result . 49
C.2.1.1.4 Measured Result . 50
C.2.1.1.5 Analysis . 50
C.3 Benchmarking hypervisors . 51
C.4 Benchmark Performance Metrics . 51
C.4.1 QoS metrics . 51
C.4.1.1 Throughput . 51
C.4.1.2 Latency . 52
C.4.1.3 Frame Loss Rate . 52
C.4.1.4 Back-to-Back . 52
C.4.1.5 Packet delay variation . 52
C.4.1.6 Service Disruption Time for Fail-over Convergence . 52
C.4.2 QoE Metrics . 53
C.5 Additional Performance Metrics (white box testing) . 53
Annex D (informative): Performance evaluation of an IP edge data plane . 55
D.1 Detailed description of the System Under Test . 55
D.1.1 From CPE to core network . 57
D.1.2 From core network to CPE . 57
D.1.3 Routing . 58
D.2 Test environment . 59
D.2.1 HW and SW used for the test with the older processor generation . 59
D.2.2 HW and SW used for the tests with the newer processor generation . 59
D.2.3 Test configuration. 60
D.3 Results . 61
D.3.1 Results with the older processor generation . 61
D.3.1.1 Bare Metal scenario . 61
D.3.1.2 Virtualised scenario . 61
D.3.2 Results with the newer processor generation. 62
D.3.2.1 Bare Metal scenario . 62
D.3.2.2 Virtualised scenario . 63
Annex E (informative): Bibliography . 64
History . 65

ETSI

---------------------- Page: 4 ----------------------
5 ETSI GS NFV-PER 001 V1.1.1 (2014-06)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Group Specification (GS) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "may not", "need", "need not", "will",
"will not", "can" and "cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms
for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI

---------------------- Page: 5 ----------------------
6 ETSI GS NFV-PER 001 V1.1.1 (2014-06)
1 Scope
The present document provides a list of features which the performance and portability templates (Virtual Machine
Descriptor and Compute Host Descriptor) should contain for the appropriate deployment of Virtual Machines over a
Compute Host (i.e. a "telco datacentre").
In addition, the document provides a set of recommendations and best practises on the minimum requirements that the
HW and hypervisor should have for a "telco datacentre" suitable for data-plane workloads. The recommendations and
best practises are based on tests results from the performance evaluation of data-plane workloads. It is recognized that
the recommendations are required for VNFs supporting data plane workloads and that a small portion of the
recommended list are not required in all cases of VNFs, such as VNFs related to control plane workloads.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
[1] ETSI GS NFV 003: "Network Functions Virtualisation (NFV); Terminology for Main Concepts in
NFV".
[2] ETSI GS NFV 001: "Network Functions Virtualisation (NFV); Use Cases".
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] Open Virtualisation Format Specification Version 2.1.0.
NOTE: Available at: http://www.dmtf.org/sites/default/files/standards/documents/DSP0243_2.1.0.pdf.
[i.2] Libvirt - The Virtualisation API.
NOTE: Available at: http://libvirt.org/.
®
[i.3] AWS CloudFormation.
NOTE 1: Available at: http://aws.amazon.com/cloudformation/.
®
NOTE 2: AWS CloudFormation is an example of a suitable product available commercially. This information is
given for the convenience of users of the present document and does not constitute an endorsement by
ETSI of this product.
[i.4] Portable Hardware Locality.
NOTE: Available at: http://www.open-mpi.org/projects/hwloc/.
ETSI

---------------------- Page: 6 ----------------------
7 ETSI GS NFV-PER 001 V1.1.1 (2014-06)
[i.5] IETF RFC 2544: "Benchmarking Methodology for Network Interconnect Devices".
NOTE: Available at: http://www.ietf.org/rfc/rfc2544.txt.
[i.6] IETF RFC 2889: "Benchmarking Methodology for LAN Switching Devices".
NOTE: Available at: http://www.ietf.org/rfc/rfc2889.txt.
[i.7] IETF RFC 3918: "Methodology for IP Multicast Benchmarking".
NOTE: Available at: http://www.ietf.org/rfc/rfc3918.txt.
[i.8] IETF RFC 3511: "Benchmarking Methodology for Firewall Performance".
NOTE: Available at: http://tools.ietf.org/rfc/rfc3511.txt.
[i.9] IEEE 1588: "IEEE Standard for a Precision Clock Synchronization Protocol for Networked
Measurement and Control Systems".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document the terms and definitions given in GS NFV 003 [1] and the following apply:
NOTE: A term defined in the present document takes precedence over the definition of the same term, if any, in
GS NFV 003 [1].
Network Function (NF): A functional building block within a network infrastructure, which has well-defined external
interfaces and a well-defined functional behaviour. In practical terms, a Network Function is today often a network
node or physical appliance.
Network Functions Virtualisation Infrastructure (NFVI): The NFV-Infrastructure is the totality of all hardware and
software components which build up the environment in which VNFs are deployed. The NFV-Infrastructure can span
across several locations, i.e. multiple N-PoPs. The network providing connectivity between these locations is regarded
to be part of the NFV-Infrastructure. The NFV-Infrastructure includes resources for computation, networking and
storage.
Compute Host: A Compute Host is the whole server entity, part of an NFVI, composed of a HW platform (processor,
memory, I/O devices, internal disk) and a hypervisor running on it.
Compute Host Descriptor (CHD): A Compute Host Descriptor is a template to define a storage schema for the
capabilities and up-to-date available resources which can be offered by a Compute Host server to VM Images at
deployment time. Therefore, there will be one Compute Host Descriptor for every Compute Host, containing both its
capabilities and its available resources.
Virtualised Network Function (VNF): An implementation of an NF that can be deployed on a Network Function
Virtualisation Infrastructure (NFVI). A VNF can be deployed as a set of Virtual Machines (VM), as SW components
deployable, maintainable and manageable, emulating a single computer.
Virtual Machine Instance (VM Instance): A Virtual Machine Instance is a Virtual Machine already running in a
Compute Host.
Virtual Machine Image (VM Image): A VM Image is an executable SW component, subject to be deployed in
Compute Hosts as one or several Virtual Machine Instances.
Virtual Machine Configuration (VM Configuration): A VM Configuration is the final configuration to be applied
when deploying a VM Image on a specific Compute Host.
ETSI

---------------------- Page: 7 ----------------------
8 ETSI GS NFV-PER 001 V1.1.1 (2014-06)
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
AAA Authentication, Authorization, and Accounting
API Application Programming Interface
ARP Address Resolution Protocol
AS Application Server
BBU Base Band Unit
BFD Bidirectional Forwarding Detection
BGP Border Gateway Protocol
BIOS Basic Input/Output System
BNG Broadband Network Gateway
BRAS Broadband Remote Access Server
BW Bandwidth
CDN Content Delivery Network
CGNAT Carrier Grade Network Address Translation
CHD Compute Host Descriptor
CIFS Common Internet File System
COTS Commercial Off-The-Shelf
CPE Customer Premises Equipment
CPU Central Processing Unit
C-RAN Cloud-Radio Access Network
CVLAN Customer VLAN
DCB Data Center Bridging
DDoS Distributed Denial of Service
DDR2 Double Data Rate type 2
DDR3 Double Data Rate type 3
DHCP Dynamic Host Configuration Protocol
DMA Direct Memory Access
DPI Deep Packet Inspection
DSLAM Digital Subscriber Line Access Multiplexer
DUT Device Under Test
E-CPE Enterprise-Customer Premises Equipment
ERPS Ethernet Ring Protection Switching
FFT Fast Fourier Transform
FIB Forwarding Information Base
FTP File Transfer Protocol
FW Firewall
GB GigaByte
GE Gigabit Ethernet
GGSN Gateway GPRS Support Node
GPRS General Packet Radio Service
GPS Global Positioning System
GRE Generic Routing Encapsulation
GUI Graphical User Interface
GW Gateway
HTML HyperText Markup Language
HTTP HyperText Transfer Protocol
HW Hardware
I/O Input/Output
I-CSCF Interrogating-Call Session Control Function
IMS IP Multimedia Subsystem
IO Input Output
IOMMU Input/Output Memory Management Unit
IOTLB I/O Translation Lookaside Buffer
IP Internet Protocol
IPC Inter-Process Communication
IPoE IP over Ethernet
IPsec IP security
ISIS Intermediate System to Intermediate System
ETSI

---------------------- Page: 8 ----------------------
9 ETSI GS NFV-PER 001 V1.1.1 (2014-06)
KPI Key Performance Indicator
L4 Layer 4
L7 Layer 7
LPM Longest Prefix Match
MAC Media Access Control
MAN Metropolitan Area Network
MANO MANagement and Orchestration
MGCF Media Gateway Controller Function
MME Mobility Management Entity
MMU Memory Management Unit
MOS Mean Opinion Score
MOS-AV MOS-Audio & Video
MPLS Multi-Protocol Label Switching
MSE Mean Square Error
N/A Not Applicable
NAS Network-Attached Storage
NAT Network Address Translation
NF Network Function
NFV Network Functions Virtualisation
NFVI Network Functions Virtualisation Infrastructure
NIC Network Interface Card
NUMA Non-Uniform Memory Access
OLT Optical Line Terminal
ONT Optical Network Terminal
ONU Optical Network Unit
OS Operating System
OSPF Open Shortest Path First
OVF Open Virtualisation Format
P2P Peer-to-Peer
PCI Peripheral Component Interconnect
PCIe PCI Express
PCIe VF PCIe Virtual Function
PCI-SIG PCI Special Interest Group
PCRF Policy and Charging Rules Function
P-CSCF Proxy-Call Session Control Function
PDN Packet Data Network
PE Provider Edge
P-GW PDN-Gateway
PNF Physical Network Function
PPP Point-to-Point Protocol
PPPoE Point-to-Point Protocol over Ethernet
PSNR Peak Signal-to-Noise Ratio
QoE Quality of Experience
QoS Quality of Service
R/W Read/Write
RADIUS Remote Authentication Dial In User Service
RAM Random Access Memory
RAN Radio Access Network
RDMA Remote Direct Memory Access
RGW Residential Gateway
RIB Routing Information Base
RMS Root Mean of Squares
RoCE RDMA over Converged Ethernet
RX Reception
SAN Storage Area Network
S-CSCF Serving-Call Session Control Function
SGSN Serving GPRS Support Node
S-GW Serving-Gateway
SLA Service Level Agreement
SMT Simultaneous Multi-Threading
SR-IOV Single Root I/O Virtualisation
SSIM Structural Similarity
ETSI

---------------------- Page: 9 ----------------------
10 ETSI GS NFV-PER 001 V1.1.1 (2014-06)
STB Set-Top-Box
STP Spanning Tree Protocol
SVLAN Service VLAN
SW Software
TCP Transmission Control Protocol
TLB Translation Lookaside Buffer
TWAMP Two-Way Active Measurement Protocol
TX Transmission
VF Virtual Function
VIA Virtual Interface Architecture
VIM Virtualised Infrastructure Manager
VLAN Virtual Local Area Network
VM Virtual Machine
VNF Virtualised Network Function
VQM Video Quality Metric
VTA Virtual Test Appliance
EMS Element Management System
KVM Kernel-based Virtual Machine
DNS Domain Name System
NTP Network Time Protocol
SSH Secure SHell
NFS Network File System
TA Test Agent
ISV Independent Software Vendor
ABR Available Bit Rate
CBR Constant Bit Rate
VBR Variable Bit Rate
RSS Receive Side Scaling
4 Introduction
The present document provides a list of minimal features which the VM Descriptor and Compute Host Descriptor
should contain for the appropriate deployment of VM Images over an NFVI (i.e. a "telco datacentre"), in order to
guarantee high and predictable performance of data plane workloads while assuring their portability. In addition, the
document provides a set of recommendations on the minimum requirements which HW and hypervisor should have for
a "telco datacentre" suitable for different workloads (data-plane, control-plane, etc.) present
...