ETSI GS NFV-TST 010 V2.6.1 (2020-09)

Network Functions Virtualisation (NFV) Release 2; Testing; API Conformance Testing Specification

RGS/NFV-TST010ed261

General Information

• Status: Not Published
• Technical Committee: NFV TST - Testing, Implementation, and Open Source Working Group
• Current Stage: 12 - Completion
• Due Date: 11-Sep-2020
• Completion Date: 04-Sep-2020

Standards Content (Sample)

ETSI GS NFV-TST 010 V2.6.1 (2020-09)






GROUP SPECIFICATION
Network Functions Virtualisation (NFV) Release 2;
Testing;
API Conformance Testing Specification
Disclaimer
The present document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.


---------------------- Page: 1 ----------------------
2 ETSI GS NFV-TST 010 V2.6.1 (2020-09)



Reference
RGS/NFV-TST010ed261
Keywords
API, conformance, NFV, testing
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2020.
All rights reserved.

DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.

3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 7
3 Definition of terms, symbols and abbreviations . 7
3.1 Terms . 7
3.2 Symbols . 7
3.3 Abbreviations . 7
4 Methodology . 8
4.1 General . 8
4.2 System Under Test (SUT) . 8
4.3 Test configurations . 8
4.3.1 General . 8
4.3.2 Confi g_prod_VE . 9
4.3.3 Config_prod_VNFM . 10
4.3.4 Config_prod_NFVO . 10
4.3.5 Config_prod_VNFM_GRANT . 11
4.4 Void . 11
4.5 Generic Test Description . 12
4.5.1 General . 12
4.5.2 Test Description format . 12
4.5.3 Scope of the tests . 13
4.5.3.1 General . 13
4.5.3.2 General characteristics of the reference points . 14
4.5.3.3 Basic behaviours of the API producer/consumer and verification steps . 15
4.5.3.3.0 Introduction . 15
4.5.3.3.1 Producer sends event triggered notifications based on consumer subscriptions . 15
4.5.3.3.2 Producer sends periodic notifications based on the consumer subscriptions . 17
4.5.3.3.3 Producer executes the requested API operation . 18
4.5.3.3.4 Consumer fetches the files/package info . 19
4.5.3.4 Workflow test considerations . 21
4.5.4 Verification . 21
4.5.4.0 Introduction . 21
4.5.4.1 Common verification aspects . 21
4.5.4.2 Verification aspects for individual API . 22
4.5.4.3 Verification aspects not considered . 23
5 Void . 23
6 Void . 23
7 Void . 23
Annex A (informative): Known Issues . 24
Annex B (informative): Workflow Test Descriptions . 25
Annex C: Void . 26
Annex D (normative): Word format presentation of the test suite for the Os-Ma-Nfvo
Reference Point . 27
ETSI

---------------------- Page: 3 ----------------------
4 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
Annex E (normative): Word format presentation of the test suite for the Ve-Vnfm
Reference Point . 28
Annex F (normative): Word format presentation of the test suite for the Or-Vnfm
Reference Point . 29
Annex G (informative): Change History . 30
History . 31


ETSI

---------------------- Page: 4 ----------------------
5 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Group Specification (GS) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
The interoperability among the functional entities which supports the requirements and functionalities specified in ETSI
NFV deliverables is one of the key and most important aspects to develop and deploy the NFV environment. In order to
achieve such interoperability, ETSI GR NFV-TST 007 [i.1] specifies the methodologies and test scenario for the testing
of interoperability mainly based on the ETSI NFV IFA specification series which specifies the reference point/interface
and functional requirements. At the same time, the validation of the NFV specification compliance of each functional
entities is also key aspects to be ensured for the interoperability, in particular protocol solution/API level. Therefore the
present document specifies the API conformance testing.

ETSI

---------------------- Page: 5 ----------------------
6 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
1 Scope
Scope of API conformance is the functionality test in an automated way for ETSI NFV APIs.
The goal of the present document is to specify the methodologies of conformance test including Test Descriptions for
NFV implementations with interfaces specified in the following NFV specifications: ETSI GS NFV-SOL 002 [2] for
the Ve-Vnfm reference point, ETSI GS NFV-SOL 003 [1] for the Or-Vnfm reference point and ETSI
GS NFV-SOL 005 [3] for the Os-ma-nfvo reference point.
Each ETSI NFV SOL deliverable specifies a set of interfaces built on the RESTful approach and meant to be used over
the HTTP protocol. The aim of the present document is to define the methodologies and the procedures with Test
Descriptions to test conformance of the exchanged HTTP payloads and the implementation of required actions for one
or more of the available interfaces within a reference point.
Since the targets of the testing are functionality (semantic checks) and the HTTP payloads (syntax checks),
methodologies, including test suite(s) and/or any technologies from any organizations (in particular Open Source
Initiatives) that can improve or help the (automated) test execution are also considered as being in the scope of the
present document.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference/.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI GS NFV-SOL 003 (V2.6.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; RESTful protocols specification for the Or-Vnfm Reference Point".
[2] ETSI GS NFV-SOL 002 (V2.6.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; RESTful protocols specification for the Ve-Vnfm Reference Point".
[3] ETSI GS NFV-SOL 005 (V2.6.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; RESTful protocols specification for the Os-Ma-nfvo Reference Point".
[4] ETSI GS NFV-TST 002 (V1.1.1): "Network Functions Virtualisation (NFV); Testing
Methodology; Report on NFV Interoperability Testing Methodology".
[5] ETSI GS NFV-SOL 013 (V2.6.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; Specification of common aspects for RESTful NFV MANO APIs".
ETSI

---------------------- Page: 6 ----------------------
7 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GR NFV-TST 007 (V2.5.1): "Network Functions Virtualisation (NFV) Release 2; Testing;
Guidelines on Interoperability Testing for MANO".
[i.2] Robot Framework.
NOTE: Available at http://robotframework.org.
[i.3] Robot2doc tool.
NOTE: Available at https://forge.etsi.org/rep/forge-tools/robot2doc.
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
Test Description (TD): set of information required to define/run the API conformance test and to realize the verdict for
the API conformance test
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Programming Interface
AUT API Under Test
EM Element Manager
FUT Function Under Test
IUT Implementation Under Test
LCM Life Cycle Management
LCOG Lifecycle Operation Granting
NFVI NFV Infrastructure
NFVO NFV Orchestrator
NS Network Service
NSD Network Service Destriptor
OCC OCCurrence
PM Performance Management
PNF Physical Network Function
PNFD PNF Descriptor
SUT System Under Test
TD Test Description
TLS Transport Layer Security
ETSI

---------------------- Page: 7 ----------------------
8 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
VE Virtual Element
VIM Virtualised Infrastructure Manager
VNF Virtual Network Function
VNFD VNF Descriptor
VNFM VNF Manager
VRQAN Virtualised Resources Quota Available Notification
4 Methodology
4.1 General
The purpose of general conformance testing is to determine to what extent a single implementation of a particular
standard conforms to the individual requirements of that standard. Concepts from ETSI GS NFV-TST 002 [4] are used
in the present document.
The important factors which characterize conformance testing are as follows:
• the System or Implementation Under Test (SUT or IUT) defines the boundaries (open interfaces) for testing;
• the conformance test system is a specialized tool (system) built for the purpose of testing and on which
specific test scripts can be run;
• the SUT comes from a single supplier (or, at least, a single product line);
• the tests are executed by a dedicated test system that has full control of the SUT and the ability to observe all
communications from the SUT;
• the tests are performed at open standardized interfaces that are not (usually) accessible to a normal user (i.e.
they are specified at the protocol level);
• the tests are specified at the detailed protocol level and are not usually based on functionality as experienced
by a user;
• the tests verify response or related request operation from SUT.
4.2 System Under Test (SUT)
The system under test is identified by an implementation of the function under test producing or consuming the API
under test e.g. in the case of the Or-vnfm reference point the function under test may be either a NFVO implementation
or a VNFM implementation.
The function shall be tested in isolation with respect to other functional blocks in a NFV platform, to guarantee that the
outcomes of the conformance tests are not result of interoperability issues with other components.
4.3 Test configurations
4.3.1 General
In accordance with clause 1, the scope of the present document is to define a testing methodology and test suite for both
the conformant protocol exchange (i.e. valid serialization and order of messages) and the initialization or execution of
the functionalities mandated for each protocol operation, including the conformant management of internal state.
In order to enable the FUT to correctly execute the operations mandated the FUT shall be tested while being executed in
a test environment (TSTENV) which provides all the functional elements needed for the correct outcome of the
operation.
NOTE: For example, to correctly execute an instantiation a VNFM requires evaluation in a test environment
which provides a VIM and NFVI plus the NFVO to grant the operation.
ETSI

---------------------- Page: 8 ----------------------
9 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
The test system shall provide the implementation of an API Consumer and a Notification Endpoint for the API Under
Test (AUT). Moreover, the test configuration may contain observation interfaces between the Test System and the FUT
or any other functional block which is part of the test environment. The specification of the mentioned observation
interfaces is out of the scope of the present document.
Stimuli to the FUT shall be injected by the Test System via the AUT only.
Conformance checks on the status and outcome of the operations triggered by the protocol shall be verified by the Test
System by means of:
• read operations issued via the AUT; or
• reception of notifications on the Notification Endpoint exposed by the test system; or
• other test interfaces to support triggers or verifications.
Test system
Test Environment
API
FUT
Consumer
AUT

Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.1-1: Generic SUT configuration
The test configurations specified in clause 4.3 fulfil the needs of Test Descriptions specified in annexes D, E and F
contained in archive gs_nfv-tst010v020601p0.zip which accompanies the present document for the different FUTs and
AUTs in scope of the present document.
4.3.2 Config_prod_VE
The configuration config_prod_VE shall be implemented to test APIs which are produced by FUTs in a VNF or EM.
The test environment of the VNF/EM is the NFVI where the test is executed.
ETSI

---------------------- Page: 9 ----------------------
10 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
Test system
Test Environment
API
VNF/EM
Consumer
AUT

Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.2-1: Configuration for tests of APIs with the FUTs as Producer run in a VNF/EM
4.3.3 Config_prod_VNFM
The configuration config_prod_VNFM shall be implemented to test APIs produced by FUTs which implement a
VNFM. The test environment of the virtual element is the NFVI where the VE is executed.
Test system
Test Environment
API
VNFM
Consumer
AUT

Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.3-1: Configuration for tests of APIs with VNFM as Producer
4.3.4 Config_prod_NFVO
The configuration config_prod_NFVO shall be implemented to test APIs produced by FUTs which implement a NFVO.
The test environment of the virtual element is an NFV platform providing VNFM, VIM and NFVI.
ETSI

---------------------- Page: 10 ----------------------
11 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
Test Environment Test system
API
NFVO
Consumer
AUT

Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.4-1: Configuration for tests of APIs with NFVO as Producer
4.3.5 Config_prod_VNFM_GRANT
The configuration config_prod_VNFM_GRANT shall be implemented to test APIs produced by FUTs which
implement a VNFM for VNF LCM test cases where an Operation Grant is needed. The test environment of the virtual
element is composed by the NFVI where the VE is executed and a NFVO component exposing the VNF Lifecyle
Operation Granting API.
Test Environment
NFVO

GRANT API
Test system
API
VNFM
Consumer
AUT
Notification

Endpoint
API exchanges
API notifications
Test interface

Figure 4.3.5-1: Configuration for tests of APIs with VNFM as Producer,
where the VNFLifecycleOperationGranting API is required
4.4 Void
Void.
ETSI

---------------------- Page: 11 ----------------------
12 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
4.5 Generic Test Description
4.5.1 General
The machine readable language used for the test case implementations is the Robot Framework [i.2].
The Robot code for all test cases is available at: https://forge.etsi.org/rep/nfv/api-tests/2.6.1 and in the accompanying
Docx Annexes , which contain the tabular representations of the test descriptions. The primary source for the Test
Descriptions is the online repository at ETSI Forge.
A compliant NFV MANO API Conformance test systems shall implement the Test Descriptions as documented in the
ETSI Forge repository and in the accompanying annexes which follow the requirements set in the following clauses.
Together with the normative test descriptions, informative implementation of the tests steps is made available as well in
the form of "Keywords in Robot Framework" to facilitate adoption of the present work.
Within the Robot Framework files present in the online repository, the test descriptions (which are normative) and test
steps implementations (which are informative) can be identified by the heading under which they are grouped, as
follows:
• the normative tests descriptions are grouped under the "*** Test Cases ***" heading, while
• the informative test steps implementations are grouped under the "*** Keywords ***" heading.
NOTE: The text for the headings needs to conform to the Robot Framework language syntax.
4.5.2 Test Description format
Test Descriptions specify the information required to run an "API conformance test". Test Descriptions are defined per
test, and table 4.5.2-1 defines the elements of the Test Description.
Table 4.5.2-1: Test Description entries
Elements Definition
Test ID The Test ID identifies uniquely the test. Naming convention: the Test ID is the number of
the clause in the present the present document which includes the Test Descriptions.
Test Objective The test objective indicates clearly which requirement is intended to be tested in the test.
This part eases the understanding of the Test Description behaviour. This also eases the
identification of the requirements, which were used as a basis for the test.
Pre-conditions The pre-conditions field defines the conditions that the state of the SUT shall verify
before undergoing the actual Test Description. In the process of the test, if the
preconditions are not met, it leads to the assignment of an Inconclusive verdict. This field
may include identifiers of other Tests within the present document which may be
executed to bring the SUT into a state which verifies the mandated pre-conditions.
Reference In the reference row, the Test Description writer indicates, the clauses of NFV SOL
specification where the tested requirements are expressed. This information is critical,
because it justifies the existence and the behaviour of the Test Description.
The reference row may refer to several clauses, separated by semi-colon.
Config ID The identifier of the applicable test configuration. A test configuration defines the
components that shall be included in Test System and SUT to execute the tests and their
connections.
Applicability Only optional functions which are required for this specific Test Description are listed in
this field.
Post-conditions Definition of the events that the SUT is expected to perform or shall not perform, if any,
according to the NFV SOL specifications and following the correct execution of the
actions in the expected behaviour below. If the post-conditions concur in the definition of
the verdict of the test, a step in the Test Case shall be explicitly provided.
Multiple post conditions are separated by semi-colon which expresses a logical AND
relation.
ETSI

---------------------- Page: 12 ----------------------
13 ETSI GS NFV-TST 010 V2.6.1 (2020-09)
Elements Definition
Test Case A set of steps of the test, carried out by the Test System to verify conformance of the
SUT to the Test Objective.
The steps are written in the form of Keywords in the Robot Framework language [i.2].
The verdict of the Test shall be se to 'Pass' if all the applicable steps in the Test Case
are successfully carried out and meet expected results criteria, it shall be set to 'Fail' if
any of the steps fails. The Inconclusive verdict applies in all other cases.

The Test Descriptions are formatted as tables as shown in the example shown in table 4.5.2-2.
Table 4.5.2-2: Test Description entries examples
Test Purpose
Test ID A.B.C.D.E
Test Objective Example Test Objective.
Pre-conditions Precondition example.
Reference Reference 1; reference 2.
Config ID Configuration_example_id
Applicability
Applicability statement example.
Post-conditions
Precondition 1; Precondition 2; P
...