ETSI GS NFV-TST 002 V1.1.1 (2016-10)

Network Functions Virtualisation (NFV); Testing Methodology; Report on NFV Interoperability Testing Methodology

DGS/NFV-TST002

General Information

• Status: Published
• Publication Date: 20-Oct-2016
• Technical Committee: NFV TST - Testing, Implementation, and Open Source Working Group
• Current Stage: 12 - Completion
• Due Date: 28-Oct-2016
• Completion Date: 21-Oct-2016

Standards Content (Sample)

ETSI GS NFV-TST 002 V1.1.1 (2016-10)






GROUP SPECIFICATION
Network Functions Virtualisation (NFV);
Testing Methodology;
Report on NFV Interoperability Testing Methodology
Disclaimer
The present document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

---------------------- Page: 1 ----------------------
2 ETSI GS NFV-TST 002 V1.1.1 (2016-10)



Reference
DGS/NFV-TST002
Keywords
interoperability, NFV, testing, methodology

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2016.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI GS NFV-TST 002 V1.1.1 (2016-10)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Executive summary . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions and abbreviations . 7
3.1 Definitions . 7
3.2 Abbreviations . 7
4 Interoperability Testing Methodology Guidelines for NFV . 8
4.1 Introduction . 8
4.2 Basic concepts for interoperability testing . 9
4.2.1 Overview . 9
4.2.2 System Under Test (SUT) . 9
4.2.3 Function Under Test (FUT) . 10
4.2.4 Test interfaces . 10
4.2.5 Test Environment . 10
4.2.6 Test Descriptions . 10
4.2.7 Test drivers . 11
4.3 Interoperability Test Specifications . 11
4.3.1 Overview . 11
4.3.2 Generic SUT Architecture . 11
4.3.3 Interoperable Features Statement (IFS) . 12
4.3.4 SUT Configurations . 13
4.3.5 Test Suite Structure . 13
4.3.6 Test Purposes . 14
4.3.7 Test Descriptions . 14
4.4 Interoperability Testing Process . 17
5 NFV SUT Architecture . 18
5.1 NFV Generic SUT Architecture . 18
5.2 NFV SUT Configuration 1 . 19
5.3 NFV SUT Configuration 2a . 19
5.4 NFV SUT Configuration 2b . 20
5.5 NFV SUT Configuration 3 . 21
5.6 NFV SUT Configuration 4 . 22
6 NFV Interoperability Features . 23
6.1 VNF Package Management . 23
6.1.1 Description . 23
6.1.2 SUT Configuration . 23
6.1.3 Observed Interfaces . 23
6.1.4 Test Interfaces . 24
6.2 Software Image Management . 24
6.2.1 Description . 24
6.2.2 SUT Configuration . 24
6.2.3 Observed Interfaces . 24
6.2.4 Test Interfaces . 25
6.3 VNF Lifecycle Management . 25
6.3.1 Description . 25
6.3.2 SUT Configuration . 25
ETSI

---------------------- Page: 3 ----------------------
4 ETSI GS NFV-TST 002 V1.1.1 (2016-10)
6.3.3 Observed Interfaces . 25
6.3.4 Test Interfaces . 28
6.4 VNF Configuration Management . 31
6.4.1 Description . 31
6.4.2 SUT Configuration . 31
6.4.3 Observed Interfaces . 31
6.4.4 Test Interfaces . 31
6.5 VNF Fault Management . 33
6.5.1 Description . 33
6.5.2 SUT Configuration . 33
6.5.3 Observed Interfaces . 33
6.5.4 Test Interfaces . 33
6.6 VNF Performance Management . 34
6.6.1 Description . 34
6.6.2 SUT Configuration . 34
6.6.3 Observed Interfaces . 34
6.6.4 Test Interfaces . 35
6.7 Network Service Lifecycle Management . 36
6.7.1 Description . 36
6.7.2 SUT Configuration . 37
6.7.3 Observed Interfaces . 37
6.7.4 Test Interfaces . 37
6.8 Network Service Fault Management . 39
6.8.1 Description . 39
6.8.2 SUT Configuration . 39
6.8.3 Observed Interfaces . 39
6.8.4 Test Interfaces . 39
6.9 Network Service Performance Management . 40
6.9.1 Description . 40
6.9.2 SUT Configuration . 40
6.9.3 Observed Interfaces . 40
6.9.4 Test Interfaces . 41
Annex A (informative): NFV IFS Pro-forma example . 43
Annex B (informative): Authors & contributors . 45
Annex C (informative): Bibliography . 46
History . 47

ETSI

---------------------- Page: 4 ----------------------
5 ETSI GS NFV-TST 002 V1.1.1 (2016-10)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Group Specification (GS) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Executive summary
The present document studies how interoperability test methodology can be applied to NFV by analysing some of the
core NFV capabilities and the interactions between the functional blocks defined within the NFV architectural
framework required to enable them.
Introduction
The present document provides methodology guidelines for interoperability testing of NFV features, starting from a
review of some basic concepts for interoperability testing and their fit in an NFV environment and a methodology for
the development of interoperability test specifications illustrated with examples of basic NFV operations. A high level
analysis of some core NFV capabilities allows to identify a generic architecture for the associated System Under Test
configurations, and to classify some initial Interoperability Feature areas.
The present document is organized as follows:
• Clause 4 provides an overview of common interoperability concepts and testing methodology guidelines.
• Clause 5 identifies a generic system under test (SUT) architecture and some initial SUT configurations for
interoperability testing of basic NFV capabilities.
• Clause 6 identifies and analyses some initial NFV interoperability feature areas and outlines for each of them
the impacted functional blocks and interfaces, as well as the applicable SUT configurations described in
clause 5.
ETSI

---------------------- Page: 5 ----------------------
6 ETSI GS NFV-TST 002 V1.1.1 (2016-10)
1 Scope
The present document provides some guidelines for NFV interoperability testing and identifies a generic System Under
Test (SUT) architecture for NFV, some initial SUT configurations, and some interoperability feature areas derived from
core NFV capabilities.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ISO/IEC 9646 (parts 1 to 7): "Information technology - Open Systems Interconnection -
Conformance testing methodology and framework".
[i.2] ETSI EG 202 237: "Methods for Testing and Specification (MTS); Internet Protocol Testing
(IPT); Generic approach to interoperability testing".
[i.3] ETSI EG 202 568: "Methods for Testing and Specification (MTS); Internet Protocol Testing
(IPT); Testing: Methodology and Framework". .
[i.5] ETSI GS NFV 002: "Network Functions Virtualisation (NFV); Architectural Framework".
[i.6] ETSI GS NFV-MAN 001: "Network Functions Virtualisation (NFV); Management and
Orchestration".
[i.7] ETSI GS NFV-IFA 010 (V2.1.1): "Network Functions Virtualisation (NFV); Management and
Orchestration; Functional requirements specification".
[i.8] ETSI GS NFV-IFA 005 (V2.1.1): "Network Functions Virtualisation (NFV); Management and
Orchestration; Or-Vi reference point - Interface and Information Model Specification".
[i.9] ETSI GS NFV-IFA 006: "Network Functions Virtualisation (NFV); Management and
Orchestration; Vi-Vnfm reference point - Interface and Information Model Specification".
ETSI

---------------------- Page: 6 ----------------------
7 ETSI GS NFV-TST 002 V1.1.1 (2016-10)
[i.10] ETSI GS NFV-IFA 007: "Network Functions Virtualisation (NFV); Management and
Orchestration; Or-Vnfm reference point - Interface and Information Model Specification".
[i.11] ETSI GS NFV-IFA 008: "Network Functions Virtualisation (NFV); Management and
Orchestration; Ve-Vnfm reference point - Interface and Information Model Specification".
[i.12] ETSI GS NFV-IFA 013: "Network Functions Virtualisation (NFV); Management and
Orchestration; Os-Ma-Nfvo reference point - Interface and Information Model Specification".
[i.13] ETSI GS NFV 003: "Network Functions Virtualisation (NFV); Terminology for main concepts in
NFV".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the terms and definitions given in ETSI GS NFV 003 [i.13] apply.
3.2 Abbreviations
For the purposes of the present document, the terms and definitions given in ETSI GS NFV 003 [i.13] and the following
apply:
API Application Programming Interface
CON CONformance
DUT Device Under Test
FUT Function Under Test
IFS Interoperable Features Statement
IOP InterOPerability
IUT Implementation Under Test
LCM Life Cycle Management
MMI Man-Machine Interface
NSD Network Service Descriptor
OSS Operation System Support
PICS Protocol Implementation Conformance Statement
QE Qualified Equipment
QF Qualified Function
SUT System Under Test
TD Test Description
TSS Test Suite Structure
VNFFG Virtual Network Function Forwarding Graph
ETSI

---------------------- Page: 7 ----------------------
8 ETSI GS NFV-TST 002 V1.1.1 (2016-10)
4 Interoperability Testing Methodology Guidelines for
NFV
4.1 Introduction
Well established test methodology like ETSI EG 202 237 [i.2] and ETSI EG 202 568 [i.3] describe two main and
complementary ways of testing devices implementing standardized services, which each have benefits and limitations:
• Conformance Testing can show that a product correctly implements a particular standard, that is, it
establishes whether or not the Implementation Under Test (IUT) meets the requirements specified by the
standard. For example, it will test protocol message contents and format as well as the permitted sequences of
messages. In this context:
- There is only one Implementation Under Test, which is part of the System Under Test.
- Tests are performed at open standardized interfaces which might not be accessible to an end user, and
executed by a dedicated test system that has full control of the System Under Test and the ability to
observe all incoming and out coming communications.
- The high degree of control of the test system over the sequence and contents of the protocol messages
allows to test both valid and invalid behaviour.

Figure 1: Conformance testing
• Interoperability Testing can demonstrate that a product will work with other like products: it proves that
end-to-end functionality between (at least) two functions is as required by the standard(s) on which those
functions are based. In this context:
- The System Under Test (SUT) is made of the combination of different Functions Under Test (FUT)
coming from different suppliers.
- Interoperability tests are based on functionality as experienced by a user, where the user may be human
or a software application.
- Tests are performed and observed at functional interfaces such as Man-Machine Interfaces (MMIs),
protocol service interfaces and Application Programming Interfaces (APIs).
- Testing at functional interfaces implies that interoperability tests can only describe functional behaviour
and sometimes it might not be possible to trigger or test protocol error behaviour on the interface(s)
among the FUTs.

Figure 2: Interoperability testing
NOTE: The concept of Function Under Test used in the present document corresponds to the concept of Device
Under Test (DUT) introduced in ETSI EG 202 568 [i.3].
ETSI

---------------------- Page: 8 ----------------------
9 ETSI GS NFV-TST 002 V1.1.1 (2016-10)
Conformance testing in conjunction with interoperability testing provides both the proof of conformance and the
guarantee of interoperation. ETSI EG 202 237 [i.2] and ETSI EG 202 568 [i.3] describe several approaches on how to
combine the two methods, the most common one being Interoperability Testing with Conformance Checks, where
reference points between the FUTs are monitored to verify the appropriate sequence and contents of protocol messages,
API calls, interface operations, etc.
Clauses 4.2 to 4.4 provide an overview of the main concepts and practices associated with interoperability testing. The
intention is to develop simple and pragmatic guidelines that can be used as a "cook-book", rather than a rigid
prescription of how to perform NFV interoperability testing.
The main areas of these guidelines are as follows:
• Definition of basic concepts.
• Instructions for the development of interoperability test specifications, including:
- Definition of a generic System Under Test (SUT) architecture.
- Identification of interoperability features.
- Specification of SUT configurations and Test Descriptions.
• Description of the interoperability testing process.
As their name implies, guidelines are only for guidance and the actual process followed should use and adapt whichever
of these guidelines are most applicable in each particular situation. In some cases this may mean the application of all
aspects.
4.2 Basic concepts for interoperability testing
4.2.1 Overview
There are a number of different terms and concepts that can be used when describing a test methodology. Clauses 4.2 to
4.4 describe the most important concepts used by these guidelines, which can been categorized either as part of the
System Under Test (SUT) or as part of the Test Environment.
Figure 3 provides an overview of these basic concepts, which are described in detail in clauses 4.2.2 to 4.2.7.

Figure 3: Illustration of basic concepts
4.2.2 System Under Test (SUT)
In the context of interoperability testing, the System Under Test (SUT) is made of a number of interacting Functions
Under Test (FUTs) coming from different suppliers.
ETSI

---------------------- Page: 9 ----------------------
10 ETSI GS NFV-TST 002 V1.1.1 (2016-10)
Depending on the complexity of the end-to-end system, the overall number of FUTs comprising the SUT, and the
interactions among them, it might be advisable to define different SUT configuration addressing specific functional
areas or groups of tests.
The first steps towards defining an Interoperability Tests Specification are identifying the Functions Under Test and
describing a generic architecture where all the required SUT configurations will fit.
4.2.3 Function Under Test (FUT)
In the context of NFV, a Function Under Test is a combination of software and/or hardware items which implement the
functionality of one or several NFV functional blocks and interact with other FUTs via one or more referen
...