ETSI GS NFV-TST 010 V2.7.1 (2021-06)

Network Functions Virtualisation (NFV) Release 2; Testing; API Conformance Testing Specification

RGS/NFV-TST010ed271

General Information

• Status: Not Published
• Technical Committee: NFV TST - Testing, Implementation, and Open Source Working Group
• Current Stage: 12 - Completion
• Due Date: 01-Jul-2021
• Completion Date: 04-Jun-2021

Standards Content (Sample)

ETSI GS NFV-TST 010 V2.7.1 (2021-06)






GROUP SPECIFICATION
Network Functions Virtualisation (NFV) Release 2;
Testing;
API Conformance Testing Specification
Disclaimer
The present document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.


---------------------- Page: 1 ----------------------
2 ETSI GS NFV-TST 010 V2.7.1 (2021-06)

Reference
RGS/NFV-TST010ed271
Keywords
API, conformance, NFV, testing

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - APE 7112B
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° w061004871

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Notice of disclaimer & limitation of liability
The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of
experience to understand and interpret its content in accordance with generally accepted engineering or
other professional standard and applicable regulations.
No recommendation as to products and services or vendors is made or should be implied.
No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law
and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness
for any particular purpose or against infringement of intellectual property rights.
In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.

Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not
limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property
rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages
for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use
of or inability to use the software.
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and
microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2021.
All rights reserved.

ETSI

---------------------- Page: 2 ----------------------
3 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 7
3 Definition of terms, symbols and abbreviations . 7
3.1 Terms . 7
3.2 Symbols . 7
3.3 Abbreviations . 7
4 Methodology . 8
4.1 General . 8
4.2 System Under Test (SUT) . 8
4.3 Test configurations . 8
4.3.1 General . 8
4.3.2 Config_prod_VE . 9
4.3.3 Config_prod_VNFM . 10
4.3.4 Config_prod_NFVO . 10
4.3.5 Config_prod_VNFM_GRANT . 11
4.3.6 Config_prod_Notif_Endpoint . 11
4.4 Void . 12
4.5 Generic Test Description . 12
4.5.1 General . 12
4.5.2 Test Description format . 12
4.5.3 Scope of the tests . 14
4.5.3.1 General . 14
4.5.3.2 General characteristics of the reference points . 15
4.5.3.3 Basic behaviours of the API producer/consumer and verification steps . 16
4.5.3.3.0 Introduction . 16
4.5.3.3.1 Producer sends event triggered notifications based on consumer subscriptions . 16
4.5.3.3.2 Producer sends periodic notifications based on the consumer subscriptions . 18
4.5.3.3.3 Producer executes the requested API operation . 19
4.5.3.3.4 Consumer fetches the files/package info . 20
4.5.3.4 Workflow test considerations . 22
4.5.4 Verification . 22
4.5.4.0 Introduction . 22
4.5.4.1 Common verification aspects . 22
4.5.4.2 Verification aspects for individual API . 23
4.5.4.3 Verification aspects not considered . 24
5 Void . 24
6 Void . 24
7 Void . 25
Annex A (informative): Known Issues . 26
Annex B (informative): Workflow Test Descriptions . 27
Annex C: Void . 28
ETSI

---------------------- Page: 3 ----------------------
4 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
Annex D (normative): Word format presentation of the test suite for the Os-Ma-Nfvo
Reference Point . 29
Annex E (normative): Word format presentation of the test suite for the Ve-Vnfm
Reference Point . 30
Annex F (normative): Word format presentation of the test suite for the Or-Vnfm
Reference Point . 31
Annex G (informative): Change History . 32
History . 33


ETSI

---------------------- Page: 4 ----------------------
5 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations
pertaining to these essential IPRs, if any, are publicly available for ETSI members and non-members, and can be
found in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to
ETSI in respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the
ETSI Web server (https://ipr.etsi.org/).
Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs,
including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not
referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become,
essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its

Members. 3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and of the 3GPP
Organizational Partners. oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and of the
®
oneM2M Partners. GSM and the GSM logo are trademarks registered and owned by the GSM Association.
Foreword
This Group Specification (GS) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
The interoperability among the functional entities which supports the requirements and functionalities specified in ETSI
NFV deliverables is one of the key and most important aspect to develop and deploy the NFV environment. In order to
achieve such interoperability, ETSI GR NFV-TST 007 [i.1] specifies the methodologies and test scenario for the testing
of interoperability mainly based on the ETSI NFV IFA specification series which specifies the reference point/interface
and functional requirements. At the same time, the validation of the NFV specification compliance of each functional
entities is also key aspects to be ensured for the interoperability, in particular protocol solution/API level. Therefore, the
present document specifies the API conformance testing.

ETSI

---------------------- Page: 5 ----------------------
6 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
1 Scope
Scope of API conformance is the functionality test in an automated way for ETSI NFV APIs.
The goal of the present document is to specify the methodologies of conformance test including Test Descriptions for
NFV implementations with interfaces specified in the following NFV specifications: ETSI GS NFV-SOL 002 [2] for
the Ve-Vnfm reference point, ETSI GS NFV-SOL 003 [1] for the Or-Vnfm reference point and ETSI
GS NFV-SOL 005 [3] for the Os-ma-nfvo reference point.
Each ETSI NFV SOL deliverable specifies a set of interfaces built on the RESTful approach and meant to be used over
the HTTP protocol. The aim of the present document is to define the methodologies and the procedures with Test
Descriptions to test conformance of the exchanged HTTP payloads and the implementation of required actions for one
or more of the available interfaces within a reference point.
Since the targets of the testing are functionality (semantic checks) and the HTTP payloads (syntax checks),
methodologies, including test suite(s) and/or any technologies from any organizations (in particular Open Source
Initiatives) that can improve or help the (automated) test execution are also considered as being in the scope of the
present document.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference/.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI GS NFV-SOL 003 (V2.7.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; RESTful protocols specification for the Or-Vnfm Reference Point".
[2] ETSI GS NFV-SOL 002 (V2.7.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; RESTful protocols specification for the Ve-Vnfm Reference Point".
[3] ETSI GS NFV-SOL 005 (V2.7.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; RESTful protocols specification for the Os-Ma-nfvo Reference Point".
[4] ETSI GS NFV-TST 002 (V1.1.1): "Network Functions Virtualisation (NFV); Testing
Methodology; Report on NFV Interoperability Testing Methodology".
[5] ETSI GS NFV-SOL 013 (V2.7.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; Specification of common aspects for RESTful NFV MANO APIs".
[6] ETSI GS NFV-SOL 001 (V2.7.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; NFV descriptors based on TOSCA specification".
ETSI

---------------------- Page: 6 ----------------------
7 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GR NFV-TST 007 (V2.5.1): "Network Functions Virtualisation (NFV) Release 2; Testing;
Guidelines on Interoperability Testing for MANO".
[i.2] Robot Framework.
NOTE: Available at http://robotframework.org.
[i.3] Robot2doc tool.
NOTE: Available at https://forge.etsi.org/rep/forge-tools/robot2doc.
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
Test Description (TD): set of information required to define/run the API conformance test and to realize the verdict for
the API conformance test
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Programming Interface
AUT API Under Test
EM Element Manager
FUT Function Under Test
IUT Implementation Under Test
LCM Life Cycle Management
LCOG Lifecycle Operation Granting
NFVI NFV Infrastructure
NFVO NFV Orchestrator
NS Network Service
NSD Network Service Descriptor
OCC OCCurrence
PM Performance Management
PNF Physical Network Function
PNFD PNF Descriptor
SUT System Under Test
TD Test Description
TLS Transport Layer Security
ETSI

---------------------- Page: 7 ----------------------
8 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
VE Virtual Element
VIM Virtualised Infrastructure Manager
VNF Virtual Network Function
VNFD VNF Descriptor
VNFM VNF Manager
VRQAN Virtualised Resources Quota Available Notification
4 Methodology
4.1 General
The purpose of general conformance testing is to determine to what extent a single implementation of a particular
standard conforms to the individual requirements of that standard. Concepts from ETSI GS NFV-TST 002 [4] are used
in the present document.
The important factors which characterize conformance testing are as follows:
• the System or Implementation Under Test (SUT or IUT) defines the boundaries (open interfaces) for testing;
• the conformance test system is a specialized tool (system) built for the purpose of testing and on which
specific test scripts can be run;
• the SUT comes from a single supplier (or, at least, a single product line);
• the tests are executed by a dedicated test system that has full control of the SUT and the ability to observe all
communications from the SUT;
• the tests are performed at open standardized interfaces that are not (usually) accessible to a normal user (i.e.
they are specified at the protocol level);
• the tests are specified at the detailed protocol level and are not usually based on functionality as experienced
by a user;
• the tests verify response or related request operation from SUT.
4.2 System Under Test (SUT)
The system under test is identified by an implementation of the function under test producing or consuming the API
under test e.g. in the case of the Or-vnfm reference point the function under test may be either a NFVO implementation
or a VNFM implementation.
The function shall be tested in isolation with respect to other functional blocks in a NFV platform, to guarantee that the
outcomes of the conformance tests are not result of interoperability issues with other components.
4.3 Test configurations
4.3.1 General
In accordance with clause 1, the scope of the present document is to define a testing methodology and test suite for both
the conformant protocol exchange (i.e. valid serialization and order of messages) and the initialization or execution of
the functionalities mandated for each protocol operation, including the conformant management of internal state.
In order to enable the FUT to correctly execute the operations mandated the FUT shall be tested while being executed in
a test environment (TSTENV) which provides all the functional elements needed for the correct outcome of the
operation.
NOTE: For example, to correctly execute an instantiation a VNFM requires evaluation in a test environment
which provides a VIM and NFVI plus the NFVO to grant the operation.
ETSI

---------------------- Page: 8 ----------------------
9 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
The test system shall provide the implementation of an API Consumer and a Notification Endpoint for the API Under
Test (AUT). Moreover, the test configuration may contain observation interfaces between the Test System and the FUT
or any other functional block which is part of the test environment. The specification of the mentioned observation
interfaces is out of the scope of the present document.
Stimuli to the FUT shall be injected by the Test System via the AUT only.
Conformance checks on the status and outcome of the operations triggered by the protocol shall be verified by the Test
System by means of:
• read operations issued via the AUT; or
• reception of notifications on the Notification Endpoint exposed by the test system; or
• other test interfaces to support triggers or verifications.
Test system
Test Environment
API
FUT
Consumer
AUT

Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.1-1: Generic SUT configuration
The test configurations specified in clause 4.3 fulfil the needs of Test Descriptions specified in annexes D, E and F
contained in archive gs_nfv-tst010v020701p0.zip which accompanies the present document for the different FUTs and
AUTs in scope of the present document.
4.3.2 Config_prod_VE
The configuration config_prod_VE shall be implemented to test APIs which are produced by FUTs in a VNF or EM.
The test environment of the VNF/EM is the NFVI where the test is executed.
ETSI

---------------------- Page: 9 ----------------------
10 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
Test system
Test Environment
API
VNF/EM
Consumer
AUT
Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.2-1: Configuration for tests of APIs with the FUTs as Producer run in a VNF/EM
4.3.3 Config_prod_VNFM
The configuration config_prod_VNFM shall be implemented to test APIs produced by FUTs which implement a
VNFM. The test environment of the virtual element is the NFVI where the VE is executed.
Test system
Test Environment
API
VNFM
Consumer
AUT

Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.3-1: Configuration for tests of APIs with VNFM as Producer
4.3.4 Config_prod_NFVO
The configuration config_prod_NFVO shall be implemented to test APIs produced by FUTs which implement a NFVO.
The test environment of the virtual element is an NFV platform providing VNFM, VIM and NFVI.
ETSI

---------------------- Page: 10 ----------------------
11 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
Test Environment Test system
API
NFVO
Consumer
AUT

Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.4-1: Configuration for tests of APIs with NFVO as Producer
4.3.5 Config_prod_VNFM_GRANT
The configuration config_prod_VNFM_GRANT shall be implemented to test APIs produced by FUTs which
implement a VNFM for VNF LCM test cases where an Operation Grant is needed. The test environment of the virtual
element is composed by the NFVI where the VE is executed and a NFVO component exposing the VNF Lifecyle
Operation Granting API.
Test Environment
NFVO

GRANT API
Test system
API
VNFM
Consumer
AUT

Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.5-1: Configuration for tests of APIs with VNFM as Producer,
where the VNFLifecycleOperationGranting API is required
4.3.6 Config_prod_Notif_Endpoint
The configuration config_prod_Notif_Endpoint shall be implemented to test any notification endpoint. The test
environment of the virtual element is an NFV platform providing an API Provider.
ETSI

---------------------- Page: 11 ----------------------
12 ETSI GS NFV-TST 010 V2.7.1 (2021-06)

Figure 4.3.6-1: Configuration for tests of APIs where the Test System is an API Provider,
and the AUT is a Notification endpoint
4.4 Void
Void.
4.5 Generic Test Description
4.5.1 General
The machine readable language used for the test case implementations is the Robot Framework [i.2].
The Robot code for all test cases is available at: https://forge.etsi.org/rep/nfv/api-tests/tree/2.7.1 and in the
accompanying Docx Annexes, which contain the tabular representations of the test descriptions. The primary source for
the Test Descriptions is the online repository at ETSI Forge.
A compliant NFV MANO API Conformance test systems shall implement the Test Descriptions as documented in the
ETSI Forge repository and in the accompanying annexes which follow the requirements set in the following clauses.
Together with the normative test descriptions, informative implementation of the tests steps is made available as well in
the form of "Keywords in Robot Framework" to facilitate adoption of the present work.
Within the Robot Framework files present in the online repository, the test descriptions (which are normative) and test
steps implementations (which are informative) can be identified by the heading under which they are grouped, as
follows:
• the normative tests descriptions are grouped under the "*** Test Cases ***" heading, while
• the informative test steps implementations are grouped under the "*** Keywords ***" heading.
NOTE: The text for the headings needs to conform to the Robot Framework language syntax.
4.5.2 Test Description format
Test Descriptions specify the information required to run an "API conformance test". Test Descriptions are defined per
test, and table 4.5.2-1 defines the elements of the Test Description.
ETSI

---------------------- Page: 12 ----------------------
13 ETSI GS NFV-TST 010 V2.7.1 (2021-06)
Table 4.5.2-1: Test Description entries
Elements Definition
Test ID The Test ID identifies uniquely the test. Naming convention: the Test ID is the number of
the clause in the present the present document which includes the Test Descriptions.
Test Objective The test objective indicates clearly which requirement is intended to be tested in the test.
This part eases the understanding of the Test Description behaviour. This also eases the
identification of the requirements, which were used as a basis for the test.
Pre-conditions The pre-conditions field defines the conditions that the state of the SUT shall verify
before undergoing the actual Test Description. In the process of the test, if the
preconditions are not met, it leads to the assignment of an Inconclusive verdict. This field
may include identifiers of other Tests within the present do
...