ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Network Functions Virtualisation (NFV); Pre-deployment Testing; Report on Validation of NFV Environments and Services

DGS/NFV-TST001

General Information

• Status: Published
• Publication Date: 03-Apr-2016
• Technical Committee: NFV TST - Testing, Implementation, and Open Source Working Group
• Current Stage: 12 - Completion
• Due Date: 01-Apr-2016
• Completion Date: 04-Apr-2016

Standards Content (Sample)

ETSI GS NFV-TST 001 V1.1.1 (2016-04)






GROUP SPECIFICATION
Network Functions Virtualisation (NFV);
Pre-deployment Testing;
Report on Validation of NFV Environments and Services
Disclaimer
The present document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

---------------------- Page: 1 ----------------------
2 ETSI GS NFV-TST 001 V1.1.1 (2016-04)



Reference
DGS/NFV-TST001
Keywords
benchmarking, NFV, testing, validation

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2016.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Abbreviations . 7
4 Definition of SUTs . 8
4.1 Overview . 8
4.2 System Under Test (SUT) . 8
4.3 Test environment . 8
4.4 Test function . 8
4.5 NFV Infrastructure Under Test . 8
4.6 VNF Under Test . 10
4.7 NS Under Test . 11
4.8 Management and Orchestration Under Test . 11
4.9 NFV Infrastructure + VIM Under Test. 12
5 Test methods for pre-deployment validation of SUTs . 14
5.1 Validating physical DUTs and SUTs . 14
5.1.1 Overview . 14
5.1.2 Data plane validation . 14
5.1.3 Control plane benchmarking . 14
5.1.4 Management plane validation - Testing fault detection, recovery and convergence . 15
5.2 Impact of virtualisation on testing methods . 15
5.3 Common test methods and specifications for virtual environments . 17
5.4 Considerations on choice of virtualised versus hardware based test appliances . 19
6 Pre-deployment validation of NFV Infrastructure . 20
6.1 Introduction . 20
6.2 Infrastructure characteristics . 21
6.3 Scenario validation . 22
6.4 Reference VNF modelling . 25
6.5 Test Case composition . 28
6.6 Method for validation . 33
7 Pre-deployment validation of VNFs . 36
7.1 VNF lifecycle testing . 36
7.1.1 Introduction. 36
7.1.2 VNF instantiation testing . 36
7.1.3 VNF instantiation in the presence of (noisy) neighbours . 39
7.1.4 VNF Scaling . 41
7.1.4.1 Introduction . 41
7.1.4.2 Metrics and Methods for validating VNF Autoscaling . 42
7.1.4.3 VNF Autoscaling validation. 44
7.1.5 VNF Termination . 48
7.2 VNF data plane benchmarking . 49
7.2.1 Introduction. 49
7.2.2 Data plane benchmarking of L2-L3 devices . 49
7.2.2.1 Introduction . 49
7.2.2.2 Forwarding Performance Benchmarking Test . 49
7.2.2.3 Long duration traffic testing . 51
7.2.2.4 IMIX Sweep Test . 52
7.2.2.5 Flow Misrouting . 53
7.2.2.6 Data Integrity Test. 54
ETSI

---------------------- Page: 3 ----------------------
4 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
7.2.3 Data plane benchmarking of L4-L7 devices . 54
7.2.3.1 Introduction . 54
7.2.3.2 VNF Application Throughput Test . 55
7.3 VNF control plane benchmarking . 56
7.3.1 Introduction. 56
7.3.2 vMME Control Plane Benchmarking . 56
7.4 VNF control & user plane benchmarking . 59
7.4.1 Introduction. 59
7.4.2 vGW's Decoupled Control and User Plane Testing . 59
8 Pre-deployment validation of Network Services . 63
8.1 Introduction . 63
8.2 Network Services -Instantiation testing . 63
8.3 Network Services - Speed of activation . 65
8.4 Network Services - Autoscaling validation . 67
Annex A (informative): Authors & contributors . 71
Annex B (informative): Change History . 72
History . 73

ETSI

---------------------- Page: 4 ----------------------
5 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Group Specification (GS) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI

---------------------- Page: 5 ----------------------
6 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
1 Scope
The present document is an informative report on methods for pre-deployment testing of the functional components of
an NFV environment. The NFV components addressed in the present document include Virtual Network Functions
(VNFs), the NFV Infrastructure (NFVI) and the NFV Management and Orchestration (NFV MANO). The
recommendations focus on lab testing and the following aspects of pre-deployment testing:
1) Assessing the performance of the NFVI and its ability to fulfil the performance and reliability requirements of
the VNFs executing on the NFVI.
2) Data and control plane testing of VNFs and their interactions with the NFV Infrastructure and the NFV
MANO.
3) Validating the performance, reliability and scaling capabilities of Network Services.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI GS NFV 003: "Network Functions Virtualisation (NFV); Terminology for Main Concepts in
NFV".
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GS NFV-SWA 001: "Network Functions Virtualisation (NFV); Virtual Network Functions
Architecture".
[i.2] IETF RFC 2544: "Benchmarking Methodology for Network Interconnect Devices".
[i.3] IETF RFC 2889: "Benchmarking Methodology for LAN Switching Devices".
[i.4] IETF RFC 5180: "IPv6 Benchmarking Methodology for Network Interconnect Devices".
[i.5] ETSI GS NFV 002: "Network Functions Virtualisation (NFV); Architectural Framework".
[i.6] ETSI GS NFV-INF 010: "Network Functions Virtualisation (NFV); Service Quality Metrics".
[i.7] ETSI GS NFV 001: "Network Functions Virtualisation (NFV); Use Cases".
[i.8] ETSI GS NFV-MAN 001: "Network Functions Virtualisation (NFV); Management and
Orchestration".
ETSI

---------------------- Page: 6 ----------------------
7 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
[i.9] ETSI GS NFV-PER 001: "Network Functions Virtualisation (NFV); NFV Performance &
Portability Best Practises".
[i.10] IETF draft-vsperf-bmwg-vswitch-opnfv-01: "Benchmarking virtual switches in OPNFV".
[i.11] IETF RFC 4656: "One Way Active Measurement Protocol".
[i.12] IETF RFC 5357: "Two Way Active Measurement Protocol".
[i.13] One-Way Active Measurement Protocol (OWAMP).
NOTE: Available at http://software.internet2.edu/owamp/.
[i.14] IETF draft-ietf-bmwg-virtual-net-01: "Considerations for Benchmarking Virtual Network
Functions and Their Infrastructure".
[i.15] IETF draft-huang-bmwg-virtual-network-performance-01: "Benchmarking methodology for
Virtualisation Network Performance".
[i.16] ETSI GS NFV-INF 004: "Network Functions Virtualisation (NFV); Infrastructure; Hypervisor
Domain".
[i.17] ETSI TS 123 002: "Digital cellular telecommunications system (Phase 2+) (GSM); Universal
Mobile Telecommunications System (UMTS); LTE; Network architecture (3GPP TS 23.002)".
[i.18] ETSI TR 121 905: "Digital cellular telecommunications system (Phase 2+) (GSM); Universal
Mobile Telecommunications System (UMTS); LTE; Vocabulary for 3GPP Specifications (3GPP
TR 21.905)".
[i.19] ETSI TS 122 278: "Universal Mobile Telecommunications System (UMTS); LTE; Service
requirements for the Evolved Packet System (EPS) (3GPP TS 22.278)".
[i.20] IETF RFC 5481: "Packet Delay Variation Applicability Statement".
[i.21] IETF RFC 6985: "IMIX Genome".
[i.22] IETF RFC 2647: "Vocabulary for 3GPP Specifications".
[i.23] IETF RFC 3511: "Service Requirements for the Evolved Packet System (EPS)".
[i.24] IETF RFC 6349: "Packet Delay Variation Applicability Statement".
[i.25] IETF RFC 7230 to IETF RFC 7239: The family of IETF RFCs that specify HTTP/1.1.
[i.26] IETF RFC 4271: "A Border Gateway Protocol 4 (BGP-4)".
[i.27] IETF RFC 2328: "OSPF Version 2".
3 Abbreviations
For the purposes of the present document, the abbreviations given in ETSI GS NFV 003 [1] and the following apply:
BFD Bidirectional Forwarding Detection
BGP Border Gateway Protocol
DoA Dead on Arrival
DUT Device Under Test
FUT Function Under Test
IMIX Internet MIX
NOTE: Some benchmarking methodologies use constant packet sizes, others use a mixture of packet sizes, or
"IMIX" ("Internet Mix").
ISIS Intermediate System to Intermediate System
LDP Label Distribution Protocol
NSUT Network Service Under Test
ETSI

---------------------- Page: 7 ----------------------
8 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
OSPF Open Shortest Path First
OWAMP One Way Active Measurement Protocol
RSVP Resource ReserVation Protocol
SUT System Under Test
TWAMP Two Way Active Measurement Protocol
VNFUT Virtual Network Function Under Test
WG Working Group
4 Definition of SUTs
4.1 Overview
All the recommended test methods (e.g. functional testing, performance testing etc.) address a certain target to be
validated and a test environment enabling the test execution. A test target in the context of the present document is
considered to be the System Under Test (SUT) which comprises one or more Functions Under Test (FUT).
The following clauses describe the general definitions of SUTs, the test environment, the test function and the NFV
components considered as SUTs for pre-deployment validation.
All descriptions provide a functional view; connections between elements in the figures 4.1, 4.2, 4.3, 4.4, 4.5 and 4.6
illustrate functional interaction.
4.2 System Under Test (SUT)
In the context of pre-deployment validation, the System Under Test (SUT) consists of one or more functions under test.
NOTE: The functions under test (FUT) are entities which are also commonly known as Devices Under Test
(DUT) in the testing community. The term Device Under Test is not used in the present document in
order to avoid ambiguities; devices are often considered to be physical entities which does not apply here.
In order to illustrate this concept, the functions under test could for example be implementations of functional blocks
from the NFV architecture such as virtualisation layer or VNF. However, other physical or virtual components could as
well be functions under test (FUT), like a virtual switch for example.
Each test specification validates one SUT where the SUT is one or more functional components of the NFV
architecture. The SUTs considered for pre-deployment validation are the NFV Infrastructure (NFVI), a Virtualised
Network Function (VNF), a Network Service (NS) or the Management and Orchestration (MANO).
It has to be noted that even though the MANO or parts of it are listed as potential SUTs, no direct pre-deployment
validation methodologies of them are in the scope of this report. However they are required as supporting functional
blocks for the validation of other entities and are listed for completeness and might be considered for further study.
4.3 Test environment
The test environment for pre-deployment validation consists of reference implementations of those functional NFV
components from the NFV architecture which do not represent the particular SUT. Additionally the test environment
contains test functions and entities to enable controlling the test execution and collecting the test measurements.
4.4 Test function
The test functions for pre-deployment validation are entities that communicate with the SUT via standardized
interfaces. The test functions are controlled from the test environment for test execution and are monitored from the test
environment to obtain measurements for test results.
4.5 NFV Infrastructure Under Test
For pre-deployment validation of the NFV Infrastructure (NFVI), the NFVI represents the SUT.
ETSI

---------------------- Page: 8 ----------------------
9 ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Figure 4.1: Functional architecture for NFVI under test
As illustrated in figure 4.1, the SUT comprises of the following functions under test (FUT):
• Physical Compute
• Physical Network
• Physical Storage
• Virtualisation Layer
• Virtual Compute
• Virtual Network
• Virtual Storage
The test environment consists of a reference implementation of the NFV MANO functional components plus a Test
Controller, Test PNFs/VNFs, Reference VNFs and a Performance Monitor. In case required for maintaining the test and
reference PNFs/VNFs, an optional Element Manager might be part of the test environment as well.
Different Reference VNFs as test functions are required to cover all aspects concerning different VNF types. The
Reference VNFs are expected to be of the types described in ETSI GS NFV-SWA 001 [i.1], annex B, and shown in
figure 4.2.
ETSI

---------------------- Page: 9 ----------------------
10 ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Figure 4.2: Reference VNF types (ETSI GS NFV-SWA 001 [i.1])
A Performance Monitor as test function is required to measure the performance indicators from the NFVI.
Optional test PNFs/VNFs might be required for certain test methods to enable traffic scenarios towards the Reference
VNFs.
4.6 VNF Under Test
For pre-deployment validation of a Virtualised Network Function (VNF), the SUT consists of one FUT which is the
VNF Under Test, see figure 4.3.

Figure 4.3: Functional architecture for VNF Under Test
The test environment consists of reference implementations of NFVI and NFV MANO functional components plus a
Test Controller and Test PNFs/VNFs. In case required for maintaining the test PNFs/VNFs and the VNF Under Test, an
optional Element Manager might be part of the test environment as well.
The Test PNFs/VNFs enable traffic scenarios towards the VNF Under Test and provide interfaces exposing access to
functional and performance indicators.
ETSI

---------------------- Page: 10 ----------------------
11 ETSI GS NFV-TST 001 V1.1.1 (2016-04)
4.7 NS Under Test
For pre-deployment validation of a Network Service (NS), the NS represents the SUT.

Figure 4.4: Functional architecture for NS Under Test
Note that in figures 4.1, 4.2, 4.3, 4.4, 4.5 and 4.6, there is a physical overlap between the SUT and the NFVI in the Test
Environment. For example, the VNF FG overlaps with the Virtual Network aspect of the NFVI.
The SUT consists of two or more VNFs and a VNF Forwarding Graph (VNF FG) which represent the Functions Under
Test respectively.
The test environment consists of reference implementations of NFVI and NFV MANO functional components plus a
Test Controller and Test PNFs/VNFs. In case required for maintaining the test PNFs/VNFs and the VNFs as FUTs of
the NS Under Test, an optional Element Manager might be part of the test environment as well.
The Test PNFs/VNFs enable traffic scenarios towards the NS Under Test and provide interfaces exposing access to
functional and performance indicators.
4.8 Management and Orchestration Under Test
For pre-deployment validation of the Management and Orchestration (MANO), the MANO represents the SUT. As
mentioned before, no direct pre-deployment validation methodologies of the MANO are in the scope of the present
document but the corresponding SUT is listed for completeness and for further studies.
ETSI

---------------------- Page: 11 ----------------------
12 ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Figure 4.5: Functional architecture for MANO Under Test
The SUT consists of the NFV Orchestrator (NFVO), the VNF Manager (VNFM) and the Virtual Infrastructure Manager
(VIM) which represent the functions under test respectively. See also figure 4.5.
The test environment consists of a reference implementation of NFVI plus a Test Controller and reference VNFs. In
case required for maintaining the reference VNFs, an optional Element Manager might be part of the test environment
as well.
Different Reference VNFs are required as test functions to cover all aspects concerning different VNF types. The
Reference VNFs are expected to be of the types as described in ETSI GS NFV-SWA 001 [i.1], annex B, and shown in
figure 4.2.
4.9 NFV Infrastructure + VIM Under Test
A variant of the NFVI Under Test could be acombination of the NFVI and the Virtual Infrastructure Manager (VIM)
Under Test. For pre-deployment validation of the NFV Infrastructure (NFVI) including the VIM, the NFVI and the
VIM represent the SUT. Even though this report does not contain direct pre-deployment validation methodologies for
this combination, it is listed for completeness and for further studies.
ETSI

---------------------- Page: 12 ----------------------
13 ETSI GS NFV-TST 001 V1.1.1 (2016-04)

Figure 4.6: Functional architecture for NFVI + VIM Under Test
As illustrated in figure 4.6, the SUT comprises of the following functions under test (FUT):
• Physical Compute
• Physical Network
• Physical Storage
• Virtualisation Layer
• Virtual Compute
• Virtual Network
• Virtual Storage
• Virtual Infrastructure Manager
The test environment consists of a reference implementation of the NFV MANO functional components excluding the
VIM plus a Test Controller, Test PNFs/VNFs, Reference VNFs and a Performance Monitor. In case required for
maintaining the test and reference PNFs/VNFs, an optional Element Manager might be part of the test environment as
well.
Different Reference VNFs as test functions are required to cover all aspects concerning different VNF types. The
Reference VNFs are expected to be of the types described in ET
...