ETSI GR NFV-TST 012 V1.1.1 (2024-02)

Network Functions Virtualisation (NFV); Testing; VIM & NFVI Control and Management Performance Evaluation

DGR/NFV-TST012

General Information

• Status: Not Published
• Technical Committee: NFV TST - Testing, Implementation, and Open Source Working Group
• Current Stage: 12 - Completion
• Due Date: 15-Mar-2024
• Completion Date: 16-Feb-2024

Standards Content (Sample)

GROUP REPORT
Network Functions Virtualisation (NFV);
Testing;
VIM & NFVI Control and Management Performance Evaluation
Disclaimer
The present document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

2 ETSI GR NFV-TST 012 V1.1.1 (2024-02)

Reference
DGR/NFV-TST012
Keywords
NFV, NFVI, performance, testing

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - APE 7112B
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° w061004871

Important notice
The present document can be downloaded from:
https://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
If you find a security vulnerability in the present document, please report it through our
Coordinated Vulnerability Disclosure Program:
https://www.etsi.org/standards/coordinated-vulnerability-disclosure
Notice of disclaimer & limitation of liability
The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of
experience to understand and interpret its content in accordance with generally accepted engineering or
other professional standard and applicable regulations.
No recommendation as to products and services or vendors is made or should be implied.
No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law
and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness
for any particular purpose or against infringement of intellectual property rights.
In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.

Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not
limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property
rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages
for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use
of or inability to use the software.
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and
microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2024.
All rights reserved.
ETSI
3 ETSI GR NFV-TST 012 V1.1.1 (2024-02)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
1 Scope . 5
2 References . 5
2.1 Normative references . 5
2.2 Informative references . 5
3 Definition of terms, symbols and abbreviations . 5
3.1 Terms . 5
3.2 Symbols . 5
3.3 Abbreviations . 6
4 Overview . 6
4.1 Introduction . 6
4.2 Background . 7
4.3 Use cases . 8
4.3.1 Use cases related to implementation selection . 8
5 Framework for Metric and Measurement Definition . 9
5.1 Performance Metric Definition Template . 9
5.2 Performance Measurement Definition T e mplate . 9
6 Test Set-ups and Configuration . 10
6.1 Test Setups . 10
6.2 Configurations . 10
7 Control and Management Performance Metric . 11
7.1 Performance metrics related to compute resource control and management . 11
7.1.1 Duration of compute resource allocation . 11
7.1.2 Duration of compute resource termination . 12
7.1.3 Duration of compute resource scaling . 13
7.1.4 Duration of compute resource migration . 14
7.2 Success rate of resource instances migration. 14
7.3 Success rate of NFVI node evacuation . 15
8 Performance Measurement . 16
8.1 Performance measurement for the duration of compute resource allocation and termination . 16
8.2 Performance measurement for compute resource scaling . 18
8.3 Performance measurement for compute resource migration . 19
8.4 Performance measurement to measure the success rate of resource instances migration . 20
8.5 Performance measurement to measure the success rate of NFVI node evacuation . 21
History . 23

ETSI
4 ETSI GR NFV-TST 012 V1.1.1 (2024-02)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations
pertaining to these essential IPRs, if any, are publicly available for ETSI members and non-members, and can be
found in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to
ETSI in respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the
ETSI Web server (https://ipr.etsi.org/).
Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs,
including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not
referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become,
essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its

Members. 3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and of the 3GPP
Organizational Partners. oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and of the ®
oneM2M Partners. GSM and the GSM logo are trademarks registered and owned by the GSM Association.
Foreword
This Group Report (GR) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI
5 ETSI GR NFV-TST 012 V1.1.1 (2024-02)
1 Scope
The present document is a group report on methods and metrics for the evaluation of VIM and NFVI control and
management performance. The evaluated NFV components in the present document include the NFV Infrastructure
(NFVI) and the Virtualised Infrastructure Manager (VIM). The evaluating area considered by the present document is
the capability of the operation and management of virtual resources which are performed by VIM and executed by
NFVI, e.g. the time for virtualization container instantiation, scaling, migration.
Based on the performance evaluation results, the present document provides guidelines for relative comparison of
different implementations of VIM and NFVI. The present document also provides advises about how to choose the
most suitable implementation in different scenarios.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] OVP Test Specifications.
[i.2] Dovetail home page.
[i.3] OPNFV Samplevnf home page.
[i.4] ETSI GR NFV 003: "Network Functions Virtualisation (NFV); Terminology for Main Concepts in
NFV".
[i.5] ETSI GS NFV-IFA 010: "Network Functions Virtualisation (NFV) Release 4; Management and
Orchestration; Functional requirements specification".
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the terms given in ETSI GR NFV 003 [i.4] the following apply:
infrastructure domain: represents the combination of VIM and NFVI
3.2 Symbols
Void.
ETSI
6 ETSI GR NFV-TST 012 V1.1.1 (2024-02)
3.3 Abbreviations
For the purposes of the present document, the abbreviations given in ETSI GR NFV 003 [i.4] and the following apply:
GB GigaByte
MB MegaByte
NUMA Non-Uniform Memory Access
4 Overview
4.1 Introduction
As described in ETSI GR NFV 003 [i.4], NFV Infrastructure (NFVI) is the key component of the NFV architecture that
encompasses the hardware and software components on which Virtual Network Functions (VNFs) are deployed. The
Virtualized Infrastructure Manager (VIM) is a key component of the NFV-MANO architectural framework which is
responsible for controlling and managing the NFV Infrastructure (NFVI) compute, storage, and network virtual
resources. The detailed functional requirements applicable to the VIM have been defined in ETSI
GS NFV-IFA 010 [i.5].
The main job of infrastructure domain performance evaluation is to measure how well the control and management
functional requirements are fulfilled. In other words, it is aimed to evaluate the control and management capability of
VIM. The primary assumption of performance evaluation is that the functional requirements have been realized
correctly according to the corresponding functional requirements specified in ETSI GS NFV-IFA 010 [i.5] NFV
standards definition. And the interoperability requirements between VIM and NFVI have been met.
Under the current logical NFV framework, the operations for controlling and managing the NFVI resource are initiated
by VIM and then executed by NFVI. So, infrastructure domain control and management performance is affected by two
main parts:
• The ability of VIM to control and manage the resources.
• The capability of NFVI to respond to the requests from VIM.
With regard to industrial implementation, VIM and NFVI are often delivered as one product bundle. As a consequence,
the two parts are closely related and therefore it is difficult to separate them from each other during the evaluation
process. Based on those considerations, VIM and NFVI are considered as a unified system under test/evaluation in the
present document.
The performance evaluation in the present document can serve the following purposes:
1) It can be used for the relative comparison of different infrastructure domain implementations. In reality,
different NFVIs can be implemented based on different virtualization technologies and different VIMs can
have different controlling and managing strategies and algorithms. Even for the same cloud management ®
platform (e.g. Openstack ), there exist different enterprise editions. There is a lack of unified and
comprehensive performance metrics and methods to find out which implementation performs better. The
present document provides the corresponding metrics and methods to measure the performance of different
infrastructure domain implementations. It can also provide some guide for infrastructure domain selection in
different scenarios. The type and size of VNFs vary in different deployment scenarios. Therefore, operators
can have different preferences about the capability of infrastructure domain. The performance evaluation in the
present document can help operators gain a comprehensive and detailed understanding about the capability of
infrastructure domain, in order to guide operators to choose the most suitable implementation for a specific
deployment scenario.
2) It can be used to describe the performance requirements for both operators and vendors. The present document
provides a set of metrics for performance evaluation. On the one hand, those metrics can be used by operators
to quantitatively describe their own performance requirements. On the other hand, vendors can offer the
benchmarking results of these metrics to demonstrate whether the infrastructure domain implementation meets
performance requirements from operators.
ETSI
7 ETSI GR NFV-TST 012 V1.1.1 (2024-02)
3) It can help to locate resource-related problems. When a failure happens to the whole system (VNF + NFVI +
NFV-MANO), it can be caused by failed resource management, wrong VNF configurations or even a
breakdown of NFV-MANO. With the help of the proposed benchmarking methods in the present document,
the operators can easily find out whether the infrastructure domain is responsible for the failure and thus
improve the efficiency of troubleshooting.
4.2 Background
For infrastructure domain implementation, there are a lot of different choices provided by companies and communities. ®
OpenStack is an open source virtualisation management platform which can support operators to deploy VNFs by
using Virtual Machines (VMs) on COTS hardware and the OPNFV community (now part of Anuket) has implemented ®
the VIM component using OpenStack . ®
NOTE: The OpenStack Word Mark and OpenStack Logo are either registered trademarks/service marks or
trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are
used with the OpenStack Foundation's permission. ETSI is not affiliated with, endorsed or sponsored by
the OpenStack Foundation, or the OpenStack community. ®
In addition to OpenStack , there are some other commercial implementations which also use VMs for VNF deployment
provided by different companies. With the development of virtualization technologies, more and more companies are
considering to build a container-based platform for NFV.
The present document provides metrics and methods which can be used to evaluate the performance of different
infrastructure domain implementations. There are some projects in communities and industries which can be as the
reference to the present document.
® ®
Project Rally in OpenStack designs the test tool to check whether OpenStack works well under high load. The
following table lists the test cases provided by Rally which are related to infrastructure domain control and management
performance evaluation.
Table 4.2-1: Reference Use Cases from Project Rally ®
OpenStack Component Test cases Corresponding requirement
Cinder Create-and-attach-volume Related to performance evaluation for VIM's
Create-and-delete-volume storage resource management
Create-and-list-volume
Create-and-update-volume
Create-and-delete-snapshot Related to performance evaluation for VIM's
Create-and-list-snapshot virtualized resource snapshot management
Swift Create-container-and-object-then-delete-all Related to performance evaluation for VIM's
Create-container-and-object-then-list-object storage resource management
Glance Create-and-deactivate-image Related to performance evaluation for VIM's
Create-and-delete-image software image management
Create-and-get-image
Create-and-list-image
Create-and-update-image
Neutron Create-and-delete-network Related to performance evaluation for VIM's
Create-and-delete-ports network resource management
Create-and-delete-routers
Create-and-delete-subnets
Create-and-show-subnets
Create-and-update-subnets
Nova Boot-and-block-migrate Related to performance evaluation for VIM's
Boot-and-delete resource instance (VM for OpenStack )
Boot-and-list management
Boot-and-live-migrate
Boot-and-migrate
The OPNFV Verified Program (OVP) [i.1] provides a series of test areas aimed to evaluate the operation of an NFV
system in accordance with carrier networking needs. OPNFV implements OVP in the Dovetail project [i.2]. Each test
area contains a number of associated test cases which are described in detail in the associated test specification. The
following table lists the test specifications for image test, VM resource scheduling on multiple nodes test and common
virtual machine life cycle events test provided by OVP.
ETSI
8 ETSI GR NFV-TST 012 V1.1.1 (2024-02)
Table 4.2-2: Reference Test Specification from Project OVP
Test Specification Test Cases Corresponding requirement
Tempest Image test specification Register Image Related to performance evaluation for
Upload Image VIM's image management
Get Image
VM Resource Scheduling on Multiple Schedule VM to compute nodes Related to performance evaluation for
Nodes test specification Create and delete multiple server VIM's resource instance (VM)
groups with same name and policy management
Create and delete server group with
affinity policy
Create and delete server group with
anti-affinity policy
List server groups
Show server group details
Common virtual machine life cycle Resize a server Related to performance evaluation for
events test specification Resizing a volume-backed server VIM's resource instance (VM)
Cold migrate a server management
Live migrate a server
The test cases provided by OVP are designed for function test of VIM and NFVI only. They can be used as the
reference for the design of test method in the present document.
4.3 Use cases
4.3.1 Use cases related to implementation selection
The use case is limited to the case that multiple infrastructure domain candidates are provided for operators to select.
They are installed in the same test environment with same set of testing VNFs. The set of testing VNFs is designed with
simple function and different resource requirements. They will be operated (instantiated, scaled, destroyed, etc.) in
order to test the control and management plane of infrastructure domain. Continuous monitoring allows the testers
(operators) to get the performance data about the infrastructure domain control and management functional behaviours
in the same operator environment.
With those performance data, operators can run relative comparison between different infrastructure domains. Those
performance data can also help operator to have comprehensive understanding about their capabilities from different
dimensions. Combining with the requirements from different VNF deployment scenarios, operator can choose the most
suitable infrastructure domain implementation.
EXAMPLE: In the scenarios which have a lot of micro-VNFs with short life cycle, operator could prefer the
implementation with better flexibility in resource control and management. The performance data
collected from the measurement can help operators to find the most suitable one.

Figure 4.3.1-1: Use Case Related to Implementation Selection
ETSI
9 ETSI GR NFV-TST 012 V1.1.1 (2024-02)
5 Framework for Metric and Measurement Definition
5.1 Performance Metric Definition Template
The present document provides a set of performance metrics definitions according to the following template, where
each performance metric is defined with the following elements:
a) Background Introduction:
- This subclause contains the background information of the performance metric.
b) Name:
- This subclause contains the name of performance metric. For the metric which may already exist in
industry, it may be renamed in this subclause.
c) Parameters:
- This subclause contains the parameters (input factors) which need to be specified in order to collect the
performance metric.
d) Unit(s):
- This subclause specifies the unit(s) of performance metric.
e) Definition:
- This subclause contains the definition of performance metric. It explains which control and management
function of VIM the metric is defined to measure.
f) Method of Measurement:
- This subclause contains the method to measure the performance metric.
5.2 Performance Measurement Definition Template
The present document provides a set of performance measurement definitions according to the following template,
where each performance measurement is defined with the following elements:
a) Description:
- This subclause contains the description of the performance measurement.
b) Measurement Name:
- This subclause contains the name which is used to identify the performance measurement.
c) Measurement Method:
- This subclause contains the methods by which the measurement is obtained.
d) Measured Metric(s):
- This subclause lists the metrics which can be measured through the performance measurement.
e) Trigger:
- This subclause contains the trigger which starts the measurement.
f) Sources of Error:
- This subclause lists the factors which may cause failure to the performance measurement.
ETSI
10 ETSI GR NFV-TST 012 V1.1.1 (2024-02)
g) Measurement Procedure:
- This subclause provides the procedure of the performance measurement.
6 Test Set-ups and Configuration
6.1 Test Setups
In the context of performance evaluation, the System Under Test (SUT) consists of infrastructure domain (VIM and
NFVI) coming from different providers.

Figure 6.1-1: System Under Test and Test Environment
As illustrated in figure 6.1-1, the test environment consists of a reference implementation of the NFV MANO functional
components (NFVO and VNFM), reference VNFs, one performance monitor and one test controller.
The testing domain is the performance of the operations and management resource instance and virtual resource. The
NFV MANO functional components (NFVO and VNFM) are responsible to help VIM to trigger the control and
management operations to be evaluated. The test controller in figure 6.1-1 is used to control the whole test procedure.
The performance monitor measures the performance indicators from the VIM. According to the definition of testing
domain, the performance of VNFs is out of the scope. During the evaluation, a set of VNFs with simple functions and
different resource requirements are used as the reference VNFs.
6.2 Configurations
The one purpose of performance measurement is to help operators select the most suitable implementation of the
infrastructure domain. The performance measurement results are intended to be used for the relative comparison of
different implementations. To ensure fairness of performance measurement, the configuration of the measured
infrastructure domain all aspects of hardware, software, and reference VNF is clearly specified.
The following hardware related configuration parameters are specified before the measurement:
1) The number of NFVI nodes: To measure the control and management capability of the infrastructure domain,
the number of NFVI nodes are specified the same for different implementations.
2) The available resource provided by each NFVI node: The number of NFVI nodes and the available resource
provided by each NFVI node are two important parameters that describe the available resource managed by
VIM. For example, ten small NFVI nodes and five big NFVI nodes require different management capabilities
from VIM, even their total amounts of available resources are equal. The available resources provided by each
NFVI node are specified by using the following configuration parameters:
- The number of available CPU cores.
- Type and size of available memory.
ETSI
11 ETSI GR NFV-TST 012 V1.1.1 (2024-02)
- Type and size of available disk storage.
- Type and number of physical NICs (The version of the NIC driver is also specified. Those parameters
can affect the forwarding performance of NFVI nodes, and thus measurement results).
- Type and size of the acceleration resource (optional).
Different implementations of infrastructure domain can have the same software dependencies, e.g. OVS. The version of
that software is specified to be the same.
One flexible infrastructure domain is expected to be able to support the deployment of different VNFs provided by
different vendors. To measure the performance of the infrastructure domain independently and objectively, the
reference VNFs are used.
For different implementations of the infrastructure domain, the reference VNFs used in performance measurement are
the same. The OPNFV Samplevnf project provides a series of sample VNFs (DPPD-PROX, UDP_Replay, vACL,
vCGNAPT and vFW) [i.3] which can be used as the reference VNFs in the performance measurement.
The resource control and management performance of the infrastructure domain varies in different VNF deployment
scenarios because different VNFs have different resource requirements (as specified in the corresponding VDU). To
measure the control and management capability of the infrastructure domain comprehensively, users define a set of
reference VNFs which have the same simple function but are based on different VDUs for the same performance
measurement. The present document provides one example set of VDUs:
Table 6.2-1: Example Set of VDUs
Small VDU Medium VDU Large VDU
CPU Cores 1 core 2 cores 4 cores
Memory 1 GB 4 GB 8 GB
Disk 10 GB 20 GB 40 GB
swImage Software images loaded on the virtualized container are the same for different VDUs.

7 Control and Management Performance Metric
7.1 Performance metrics related to compute resource control
and management
7.1.1 Duration of compute resource allocation
a) Background Introduction:
- During the instantiation of a VNF, infrastructure domains are expected to perform the operation of
compute resource allocation a
...