ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Network Functions Virtualisation (NFV); Pre-deployment Testing; Report on Validation of NFV Environments and Services

DGS/NFV-TST001

General Information

• Status: Published
• Publication Date: 03-Apr-2016
• Technical Committee: NFV TST - Testing, Implementation, and Open Source Working Group
• Current Stage: 12 - Completion
• Due Date: 01-Apr-2016
• Completion Date: 04-Apr-2016

Standards Content (Sample)

GROUP SPECIFICATION
Network Functions Virtualisation (NFV);
Pre-deployment Testing;
Report on Validation of NFV Environments and Services
Disclaimer
The present document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

2 ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Reference
DGS/NFV-TST001
Keywords
benchmarking, NFV, testing, validation

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2016.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Abbreviations . 7
4 Definition of SUTs . 8
4.1 Overview . 8
4.2 System Under Test (SUT) . 8
4.3 Test environment . 8
4.4 Test function . 8
4.5 NFV Infrastructure Under Test . 8
4.6 VNF Under Test . 10
4.7 NS Under Test . 11
4.8 Management and Orchestration Under Test . 11
4.9 NFV Infrastructure + VIM Under Test. 12
5 Test methods for pre-deployment validation of SUTs . 14
5.1 Validating physical DUTs and SUTs . 14
5.1.1 Overview . 14
5.1.2 Data plane validation . 14
5.1.3 Control plane benchmarking . 14
5.1.4 Management plane validation - Testing fault detection, recovery and convergence . 15
5.2 Impact of virtualisation on testing methods . 15
5.3 Common test methods and specifications for virtual environments . 17
5.4 Considerations on choice of virtualised versus hardware based test appliances . 19
6 Pre-deployment validation of NFV Infrastructure . 20
6.1 Introduction . 20
6.2 Infrastructure characteristics . 21
6.3 Scenario validation . 22
6.4 Reference VNF modelling . 25
6.5 Test Case composition . 28
6.6 Method for validation . 33
7 Pre-deployment validation of VNFs . 36
7.1 VNF lifecycle testing . 36
7.1.1 Introduction. 36
7.1.2 VNF instantiation testing . 36
7.1.3 VNF instantiation in the presence of (noisy) neighbours . 39
7.1.4 VNF Scaling . 41
7.1.4.1 Introduction . 41
7.1.4.2 Metrics and Methods for validating VNF Autoscaling . 42
7.1.4.3 VNF Autoscaling validation. 44
7.1.5 VNF Termination . 48
7.2 VNF data plane benchmarking . 49
7.2.1 Introduction. 49
7.2.2 Data plane benchmarking of L2-L3 devices . 49
7.2.2.1 Introduction . 49
7.2.2.2 Forwarding Performance Benchmarking Test . 49
7.2.2.3 Long duration traffic testing . 51
7.2.2.4 IMIX Sweep Test . 52
7.2.2.5 Flow Misrouting . 53
7.2.2.6 Data Integrity Test. 54
ETSI
4 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
7.2.3 Data plane benchmarking of L4-L7 devices . 54
7.2.3.1 Introduction . 54
7.2.3.2 VNF Application Throughput Test . 55
7.3 VNF control plane benchmarking . 56
7.3.1 Introduction. 56
7.3.2 vMME Control Plane Benchmarking . 56
7.4 VNF control & user plane benchmarking . 59
7.4.1 Introduction. 59
7.4.2 vGW's Decoupled Control and User Plane Testing . 59
8 Pre-deployment validation of Network Services . 63
8.1 Introduction . 63
8.2 Network Services -Instantiation testing . 63
8.3 Network Services - Speed of activation . 65
8.4 Network Services - Autoscaling validation . 67
Annex A (informative): Authors & contributors . 71
Annex B (informative): Change History . 72
History . 73

ETSI
5 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Group Specification (GS) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI
6 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
1 Scope
The present document is an informative report on methods for pre-deployment testing of the functional components of
an NFV environment. The NFV components addressed in the present document include Virtual Network Functions
(VNFs), the NFV Infrastructure (NFVI) and the NFV Management and Orchestration (NFV MANO). The
recommendations focus on lab testing and the following aspects of pre-deployment testing:
1) Assessing the performance of the NFVI and its ability to fulfil the performance and reliability requirements of
the VNFs executing on the NFVI.
2) Data and control plane testing of VNFs and their interactions with the NFV Infrastructure and the NFV
MANO.
3) Validating the performance, reliability and scaling capabilities of Network Services.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI GS NFV 003: "Network Functions Virtualisation (NFV); Terminology for Main Concepts in
NFV".
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GS NFV-SWA 001: "Network Functions Virtualisation (NFV); Virtual Network Functions
Architecture".
[i.2] IETF RFC 2544: "Benchmarking Methodology for Network Interconnect Devices".
[i.3] IETF RFC 2889: "Benchmarking Methodology for LAN Switching Devices".
[i.4] IETF RFC 5180: "IPv6 Benchmarking Methodology for Network Interconnect Devices".
[i.5] ETSI GS NFV 002: "Network Functions Virtualisation (NFV); Architectural Framework".
[i.6] ETSI GS NFV-INF 010: "Network Functions Virtualisation (NFV); Service Quality Metrics".
[i.7] ETSI GS NFV 001: "Network Functions Virtualisation (NFV); Use Cases".
[i.8] ETSI GS NFV-MAN 001: "Network Functions Virtualisation (NFV); Management and
Orchestration".
ETSI
7 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
[i.9] ETSI GS NFV-PER 001: "Network Functions Virtualisation (NFV); NFV Performance &
Portability Best Practises".
[i.10] IETF draft-vsperf-bmwg-vswitch-opnfv-01: "Benchmarking virtual switches in OPNFV".
[i.11] IETF RFC 4656: "One Way Active Measurement Protocol".
[i.12] IETF RFC 5357: "Two Way Active Measurement Protocol".
[i.13] One-Way Active Measurement Protocol (OWAMP).
NOTE: Available at http://software.internet2.edu/owamp/.
[i.14] IETF draft-ietf-bmwg-virtual-net-01: "Considerations for Benchmarking Virtual Network
Functions and Their Infrastructure".
[i.15] IETF draft-huang-bmwg-virtual-network-performance-01: "Benchmarking methodology for
Virtualisation Network Performance".
[i.16] ETSI GS NFV-INF 004: "Network Functions Virtualisation (NFV); Infrastructure; Hypervisor
Domain".
[i.17] ETSI TS 123 002: "Digital cellular telecommunications system (Phase 2+) (GSM); Universal
Mobile Telecommunications System (UMTS); LTE; Network architecture (3GPP TS 23.002)".
[i.18] ETSI TR 121 905: "Digital cellular telecommunications system (Phase 2+) (GSM); Universal
Mobile Telecommunications System (UMTS); LTE; Vocabulary for 3GPP Specifications (3GPP
TR 21.905)".
[i.19] ETSI TS 122 278: "Universal Mobile Telecommunications System (UMTS); LTE; Service
requirements for the Evolved Packet System (EPS) (3GPP TS 22.278)".
[i.20] IETF RFC 5481: "Packet Delay Variation Applicability Statement".
[i.21] IETF RFC 6985: "IMIX Genome".
[i.22] IETF RFC 2647: "Vocabulary for 3GPP Specifications".
[i.23] IETF RFC 3511: "Service Requirements for the Evolved Packet System (EPS)".
[i.24] IETF RFC 6349: "Packet Delay Variation Applicability Statement".
[i.25] IETF RFC 7230 to IETF RFC 7239: The family of IETF RFCs that specify HTTP/1.1.
[i.26] IETF RFC 4271: "A Border Gateway Protocol 4 (BGP-4)".
[i.27] IETF RFC 2328: "OSPF Version 2".
3 Abbreviations
For the purposes of the present document, the abbreviations given in ETSI GS NFV 003 [1] and the following apply:
BFD Bidirectional Forwarding Detection
BGP Border Gateway Protocol
DoA Dead on Arrival
DUT Device Under Test
FUT Function Under Test
IMIX Internet MIX
NOTE: Some benchmarking methodologies use constant packet sizes, others use a mixture of packet sizes, or
"IMIX" ("Internet Mix").
ISIS Intermediate System to Intermediate System
LDP Label Distribution Protocol
NSUT Network Service Under Test
ETSI
8 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
OSPF Open Shortest Path First
OWAMP One Way Active Measurement Protocol
RSVP Resource ReserVation Protocol
SUT System Under Test
TWAMP Two Way Active Measurement Protocol
VNFUT Virtual Network Function Under Test
WG Working Group
4 Definition of SUTs
4.1 Overview
All the recommended test methods (e.g. functional testing, performance testing etc.) address a certain target to be
validated and a test environment enabling the test execution. A test target in the context of the present document is
considered to be the System Under Test (SUT) which comprises one or more Functions Under Test (FUT).
The following clauses describe the general definitions of SUTs, the test environment, the test function and the NFV
components considered as SUTs for pre-deployment validation.
All descriptions provide a functional view; connections between elements in the figures 4.1, 4.2, 4.3, 4.4, 4.5 and 4.6
illustrate functional interaction.
4.2 System Under Test (SUT)
In the context of pre-deployment validation, the System Under Test (SUT) consists of one or more functions under test.
NOTE: The functions under test (FUT) are entities which are also commonly known as Devices Under Test
(DUT) in the testing community. The term Device Under Test is not used in the present document in
order to avoid ambiguities; devices are often considered to be physical entities which does not apply here.
In order to illustrate this concept, the functions under test could for example be implementations of functional blocks
from the NFV architecture such as virtualisation layer or VNF. However, other physical or virtual components could as
well be functions under test (FUT), like a virtual switch for example.
Each test specification validates one SUT where the SUT is one or more functional components of the NFV
architecture. The SUTs considered for pre-deployment validation are the NFV Infrastructure (NFVI), a Virtualised
Network Function (VNF), a Network Service (NS) or the Management and Orchestration (MANO).
It has to be noted that even though the MANO or parts of it are listed as potential SUTs, no direct pre-deployment
validation methodologies of them are in the scope of this report. However they are required as supporting functional
blocks for the validation of other entities and are listed for completeness and might be considered for further study.
4.3 Test environment
The test environment for pre-deployment validation consists of reference implementations of those functional NFV
components from the NFV architecture which do not represent the particular SUT. Additionally the test environment
contains test functions and entities to enable controlling the test execution and collecting the test measurements.
4.4 Test function
The test functions for pre-deployment validation are entities that communicate with the SUT via standardized
interfaces. The test functions are controlled from the test environment for test execution and are monitored from the test
environment to obtain measurements for test results.
4.5 NFV Infrastructure Under Test
For pre-deployment validation of the NFV Infrastructure (NFVI), the NFVI represents the SUT.
ETSI
9 ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Figure 4.1: Functional architecture for NFVI under test
As illustrated in figure 4.1, the SUT comprises of the following functions under test (FUT):
• Physical Compute
• Physical Network
• Physical Storage
• Virtualisation Layer
• Virtual Compute
• Virtual Network
• Virtual Storage
The test environment consists of a reference implementation of the NFV MANO functional components plus a Test
Controller, Test PNFs/VNFs, Reference VNFs and a Performance Monitor. In case required for maintaining the test and
reference PNFs/VNFs, an optional Element Manager might be part of the test environment as well.
Different Reference VNFs as test functions are required to cover all aspects concerning different VNF types. The
Reference VNFs are expected to be of the types described in ETSI GS NFV-SWA 001 [i.1], annex B, and shown in
figure 4.2.
ETSI
10 ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Figure 4.2: Reference VNF types (ETSI GS NFV-SWA 001 [i.1])
A Performance Monitor as test function is required to measure the performance indicators from the NFVI.
Optional test PNFs/VNFs might be required for certain test methods to enable traffic scenarios towards the Reference
VNFs.
4.6 VNF Under Test
For pre-deployment validation of a Virtualised Network Function (VNF), the SUT consists of one FUT which is the
VNF Under Test, see figure 4.3.

Figure 4.3: Functional architecture for VNF Under Test
The test environment consists of reference implementations of NFVI and NFV MANO functional components plus a
Test Controller and Test PNFs/VNFs. In case required for maintaining the test PNFs/VNFs and the VNF Under Test, an
optional Element Manager might be part of the test environment as well.
The Test PNFs/VNFs enable traffic scenarios towards the VNF Under Test and provide interfaces exposing access to
functional and performance indicators.
ETSI
11 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
4.7 NS Under Test
For pre-deployment validation of a Network Service (NS), the NS represents the SUT.

Figure 4.4: Functional architecture for NS Under Test
Note that in figures 4.1, 4.2, 4.3, 4.4, 4.5 and 4.6, there is a physical overlap between the SUT and the NFVI in the Test
Environment. For example, the VNF FG overlaps with the Virtual Network aspect of the NFVI.
The SUT consists of two or more VNFs and a VNF Forwarding Graph (VNF FG) which represent the Functions Under
Test respectively.
The test environment consists of reference implementations of NFVI and NFV MANO functional components plus a
Test Controller and Test PNFs/VNFs. In case required for maintaining the test PNFs/VNFs and the VNFs as FUTs of
the NS Under Test, an optional Element Manager might be part of the test environment as well.
The Test PNFs/VNFs enable traffic scenarios towards the NS Under Test and provide interfaces exposing access to
functional and performance indicators.
4.8 Management and Orchestration Under Test
For pre-deployment validation of the Management and Orchestration (MANO), the MANO represents the SUT. As
mentioned before, no direct pre-deployment validation methodologies of the MANO are in the scope of the present
document but the corresponding SUT is listed for completeness and for further studies.
ETSI
12 ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Figure 4.5: Functional architecture for MANO Under Test
The SUT consists of the NFV Orchestrator (NFVO), the VNF Manager (VNFM) and the Virtual Infrastructure Manager
(VIM) which represent the functions under test respectively. See also figure 4.5.
The test environment consists of a reference implementation of NFVI plus a Test Controller and reference VNFs. In
case required for maintaining the reference VNFs, an optional Element Manager might be part of the test environment
as well.
Different Reference VNFs are required as test functions to cover all aspects concerning different VNF types. The
Reference VNFs are expected to be of the types as described in ETSI GS NFV-SWA 001 [i.1], annex B, and shown in
figure 4.2.
4.9 NFV Infrastructure + VIM Under Test
A variant of the NFVI Under Test could be acombination of the NFVI and the Virtual Infrastructure Manager (VIM)
Under Test. For pre-deployment validation of the NFV Infrastructure (NFVI) including the VIM, the NFVI and the
VIM represent the SUT. Even though this report does not contain direct pre-deployment validation methodologies for
this combination, it is listed for completeness and for further studies.
ETSI
13 ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Figure 4.6: Functional architecture for NFVI + VIM Under Test
As illustrated in figure 4.6, the SUT comprises of the following functions under test (FUT):
• Physical Compute
• Physical Network
• Physical Storage
• Virtualisation Layer
• Virtual Compute
• Virtual Network
• Virtual Storage
• Virtual Infrastructure Manager
The test environment consists of a reference implementation of the NFV MANO functional components excluding the
VIM plus a Test Controller, Test PNFs/VNFs, Reference VNFs and a Performance Monitor. In case required for
maintaining the test and reference PNFs/VNFs, an optional Element Manager might be part of the test environment as
well.
Different Reference VNFs as test functions are required to cover all aspects concerning different VNF types. The
Reference VNFs are expected to be of the types described in ETSI GS NFV-SWA 001 [i.1], annex B, and shown in
figure 4.2.
A Performance Monitor as test function is required to measure the performance indicators from the NFVI.
Optional test PNFs/VNFs might be required for certain test methods to enable traffic scenarios towards the Reference
VNFs.
ETSI
14 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
5 Test methods for pre-deployment validation of SUTs
5.1 Validating physical DUTs and SUTs
5.1.1 Overview
This clause provides a high level description of the methods used to validate physical DUTs and SUTs (e.g. individual
or network of purpose built routers, switches and appliances) prior to their deployment in the field. Its purpose is to help
the reader understand the differences between the testing methods employed in physical and virtual/NFV environments.
Physical DUTs are traditionally validated using physical 'test devices'. The test device interoperates with the DUT in a
lab setup. The test device establishes sessions with the DUT and exchanges user plane and control plane traffic to assess
the functionality and performance of the DUT. Three representative use cases for validation of physical DUTs are
presented in clauses 5.1.2, 5.1.3 and 5.1.4.
5.1.2 Data plane validation
Standards based benchmarking methods are used to perform data plane validation of the physical DUT(s). A few of the
most significant benchmarking methods are listed below:
• IETF RFC 2544 [i.2] specifies methods to assess network interconnect devices and measures metrics such as
throughput, latency, frame loss rate, and system recovery time.
• IETF RFC 2889 [i.3] specifies methods for benchmarking of LAN switching devices and takes into
consideration flooding and MAC address learning.
• IETF RFC 5180 [i.4] extends IETF RFC 2544 [i.2] for IPv6 capable DUTs and networks.
In these benchmarking methods, a test device originates test traffic. The traffic is received, processed and forwarded by
the DUT(s) and terminated on another test device. The originating test device varies the frame sizes, burst sizes and
frame rates and the terminating test device measures metrics such as throughput, latency and frame loss rates. The
DUTs are connected to the test devices as shown in figure5.1. Each of the test devices can be physically connected to
the DUT on multiple (perhaps hundreds) of ports using a variety of speeds (1G, 10G, 40G, 100G, etc.) and a variety of
interface types (Ethernet, ATM, Fibre channel, etc.). Please refer to the IETF RFC 2544 [i.2], IETF RFC 2889 [i.3] and
IETF RFC 5180 [i.4] for a detailed explanation of the testing methods.

Figure 5.1: Test setup for data plane benchmarking of physical DUT(s)
5.1.3 Control plane benchmarking
The testing of a DUT for compliance to IETF RFC based protocols (for example IETF RFC 4271 [i.26] for BGP and
IETF RFC 2328 [i.27] for OSPFv2) is accomplished by connecting it to test devices that speak the same control plane
protocols. In figure 5.2, Test device 1 emulates a network of nodes that speak the same protocols as the DUT. Test
device 1 establishes control sessions (for e.g. BGP, OSPF, ISIS, RSVP, LDP and/or BFD) with the DUT, exchanges
routes, and defines traffic flows that are bound to these sessions. In effect, Test device 1 exchanges both control and
data plane traffic with the DUT. The DUT forwards the traffic which then terminates on Test device 2.
The Test devices benchmark the DUT by using the following methods:
• Scale up the number of control sessions between the test device and DUT to benchmark the maximum session
scale supported by DUT.
• Vary the session set up and tear down rates to assess DUT performance.
ETSI
15 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
• Verify that the DUT layers the control plane packets in the proper order (e.g. VPN traffic exchanged using
correct underlying label stack, DHCP over VPLS, etc.).
• Test device 2 originates data traffic destined for addresses advertised by Test device 1's routing sessions with
DUT. The ability of the DUT to correctly forward traffic from Test device 1 toward Test device 2 is validated.

Figure 5.2: Test setup for control plane benchmarking of physical DUT(s)
5.1.4 Management plane validation - Testing fault detection, recovery and
convergence
The fault detection, recovery, and convergence capabilities of the DUT are validated by connecting 3 Test device ports
to the DUT as shown in figure 5.3. Test devices 2 and 3 will advertise identical routes to a destination A, with Test
device 2 advertising a lower cost route. All traffic originated by Test device 1 for destination A will be forwarded by the
DUT to Test device 2 (which advertised lower cost route). This methodology is applicable for a wide range of routing
protocols that are used to exchange routes between the DUT and test devices.
• Test device 2 injects an error, such as withdraw route, break link, or BFD Stop.
• The test devices assess the DUT's ability to a) detect the fault quickly, b) install the backup route, c) stop
traffic toward Test device 2 and forward affected traffic toward Test device 3.
• The Test devices will work in synchronization to measure the fault detection and convergence times with
microsecond accuracy.
Figure 5.3: Test setup for management plane testing of physical DUT(s)
5.2 Impact of virtualisation on testing methods
To understand the impact of virtualisation on testing methods, it is instructive to revisit the NFV framework defined in
ETSI GS NFV 002 [i.5]. The physical DUTs described in the clause 5.1 are instantiated and executed as VNFs in an
NFV environment. In addition, NFV architecture defines new entities such as the NFVI and the NFV MANO and new
interfaces between the VNFs, NFVI and the NFV MANO components.
The new components and the new interfaces defined by the NFV architecture introduce new failure points and mandate
the need for additional testing methods to ensure the reliability of VNFs and services.
ETSI
16 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
NFV Management and Orchestration
Os-Ma
NFV
OSS/BSS
Orchestrator
Or-Vnfm
EM 1 EM 2 EM 3
Ve-Vnfm
Service, VNF and
VNF
Infrastructure
Manager(s)
Description
VNF 3
VNF 1 VNF 2
Vn-Nf
Vi-Vnfm
NFVI
Virtual Virtual
Virtual
Computing Storage Network
Nf-Vi Or-Vi
Virtualised
Virtualisation Layer
Infrastructure
Vl-Ha
Manager(s)
Hardware resources
Computing Storage Network
Hardware Hardware
Hardware
Execution reference points Main NFV reference points
Other reference points
Figure 5.4: The NFV architectural framework with reference points
New capabilities introduced by virtualisation that change the way systems are tested are:
• In traditional networks, NFs run on dedicated physical devices with dedicated resources. In a virtualised
environment, VNFs run in a shared compute, storage and network environment and may contend for the same
resources.
• The Virtualisation Layer, consisting of a hypervisor, OS container and/or vSwitch, abstracts the resource
details from the VNFs. The performance of the NFV Infrastructure is influenced by the type of load (network
versus IT workload, CPU intensive, memory intensive or storage intensive) and number of VNFs executing.
• Special data plane acceleration mechanisms may be used for IO intensive applications. Examples of such
mechanisms are DPDK and SR-IOV, which allow VNFs to bypass bottlenecks in the Virtualisation Layer
while transmitting and receiving packets.
• NFV allows for service chaining, where a forwarding graph may be designed to define the path a packet flow
will take from its source to its destination. The path may consist of one or multiple VNFs, which may or may
not be present on the same NFVI.
• A VNF will be instantiated with a defined amount of resources available to it. However, NFV allows for the
MANO function to dynamically modify the amount of resources allocated to a VNF, as well as instantiate
other VNFs, as the load requires.
• Failure recovery mechanisms allow for a VNF to be transferred to another NFVI or another VNF instantiated
to recover from a catastrophic error.
These new capabilities of NFV warrant new methods of testing and monitoring the network. Therefore, test plans will
be constructed to validate these new capabilities. While these concepts are largely independent of the function a VNF
accomplishes, the tests will take into account the type of function the VNF executes in order to drive the appropriate
traffic type for a test. For example, a Diameter server VNF will require different traffic input and output than a firewall
VNF. The methods described below will focus on the new capabilities introduced by NFV, while providing examples of
traffic types. However, the focus is not on the VNFs' specific functionality.
ETSI
17 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
The main variables (configuration items) in virtualised system are listed below:
• Resources allocated to the VNF: compute cores, memory allocation, etc.
• Resources allocated to the vSwitch.
• Virtualisation layer (hypervisor) used.
• The HW resources, including compute, networking (NIC) and storage used.
• The usage (or not) of data plane acceleration techniques.
• The MANO policy for scaling a VNFs resources or to instantiate new VNFs to handle increased load. The
opposite is true when the load drops and the VNFs can be scaled back.
• The presence or absence of other VNFs on the same NFVI (multi-tenancy), and the function of these VNFs.
Most of the tests below will exercise the concepts above, while having either fixed or variable values for the
configuration values. Depending on the objective of the test, some configuration values will be fixed, some may vary
per iteration of the test, and others will be measured, and will become the result of the test. The tests will also have as an
objective to discover the optimal settings of these configuration variables, for the desired performance of the system,
thus helping in dimensioning the system appropriately.
5.3 Common test methods and specifications for virtual
environments
There are multiple reasons to perform pre-deployment testing, most of which are not new to NFV:
• Feature verification.
• Regression testing when SW or HW changes are made.
• Availability and robustness verification.
However, some new concepts are introduced when performance testing is concerned. This is because the very nature of
virtualisation introduces many new variables and controls that affect performance, as listed above (multi-tenancy,
acceleration techniques, etc.). With this in mind, it leads to different means of approaching the following broad
categories of performance testing. These categories are not all-inclusive, but they include the majority of performance
testing. It is important to note that the discussion that follows applies to all types of pre-deployment testing, and not
only to performance testing. Performance testing is used as an example to illustrate the concepts.
1) Performance verification: The goal is to validate that a set of established numerical performance objectives can
be reached, under mostly fixed conditions within the SUT. This is typically done to verify that the published
performance metrics for a SUT (for example a VNF or an NFVI platform) can be met.
2) Benchmarking: This type of test is aimed at finding out the maximum performance level of a SUT with fixed
resources, conducted within an isolated test environment (ITE). It is a goal seeking test exercise, where
iterations of the test are run successively, in order to reach the maximum of the performance metric of interest.
3) Dimensioning: This type of test aims to determine the amount of infrastructure required to support a defined
set of performance metrics. The performance metrics are known, the objective is to determine the amount of
NFVI resources required to support the performance levels.
These are not really new categories of testing, but what is new to NFV is how to go about the testing. A widely adopted
strategy for performance testing of a SUT is to isolate the SUT in order to reduce the amount of variables in the test.
This makes it easier to ensure that the performance being measured is that of the SUT itself, without being influenced
by other devices, and it also makes it easier to repeat deterministic configurations and results. With dedicated HW
platforms supporting today's networking devices, it is possible to isolate the SUT effectively, and to remove all other
variables from a performance test. An example of this (from mobility architecture) is the ability to isolate an S/PGW for
performance testing by simulating the surrounding elements with test devices (the MME, the eNodeB, the PDN
elements, etc.)
ETSI
18 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
However, by the nature of the NFV architecture, it is very challenging to isolate one function as a SUT without the
presence of the other functions supporting the SUT. For example, it is not possible to test a specific VNF (as a SUT)
without having the NFVI present. Since the NFVI seriously impacts the performance of the VNF, this presents a
challenge for performance testing the VNF. In fact, in a typical NFV deployment, other VNFs can be running on the
same NFVI (multi-tenancy), which further complicates the testing.
The recommended way to approximate SUT isolation in an NFV environment is to strictly control the configuration
parameters of the NFV architecture elements that do not comprise the SUT (i.e. the test environment) while varying
configuration parameters for the SUT. This leads to two different sets of configuration parameters:
1) The fixed configuration parameters: these parameters remain constant for all iterations of a test exercise.
2) The variable configuration parameters: these can be modified between iterations of a test exercise.
The categories of performance tests above then help to define, from the total set of configuration parameters, which fall
into the fixed and variable parameters. The definition of the fixed and variable configuration parameters determine the
components that are isolated for the performance test (i.e. to isolate the SUT as much as feasible) and the test
environment. It should be noted that variable configuration parameters are only modified between test run iterations.
EXAMPLE 1: Performance verification of a VNF:
Typically, the supplier of a VNF will have a performance guarantee for the VNF under strict
conditions. The numerical guarantees are based on performance metrics such as packets/sec,
throughput, etc. while the strict conditions will define, among other para
...