ETSI GR NFV-TST 006 V1.1.1 (2020-01)

Network Functions Virtualisation (NFV); Testing; Report on CICD and Devops

DGR/NFV-TST006

General Information

• Status: Published
• Publication Date: 30-Jan-2020
• Technical Committee: NFV TST - Testing, Implementation, and Open Source Working Group
• Current Stage: 12 - Completion
• Due Date: 16-Jan-2020
• Completion Date: 31-Jan-2020

Standards Content (Sample)

ETSI GR NFV-TST 006 V1.1.1 (2020-01)






GROUP REPORT
Network Functions Virtualisation (NFV);
Testing;
Report on CICD and Devops
Disclaimer
The present document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

---------------------- Page: 1 ----------------------
2 ETSI GR NFV-TST 006 V1.1.1 (2020-01)



Reference
DGR/NFV-TST006
Keywords
CI/CD, DevOps, NFV, NFVI, SDN

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2020.
All rights reserved.

DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.

3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI GR NFV-TST 006 V1.1.1 (2020-01)
Contents
Intellectual Property Rights . 6
Foreword . 6
Modal verbs terminology . 6
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definition of terms, symbols and abbreviations . 8
3.1 Terms . 8
3.2 Symbols . 8
3.3 Abbreviations . 8
4 Background on Devops and CI/CD . 8
4.1 Introduction . 8
4.2 General Backgrounder . 8
4.3 DevOps in an NFV Context . 11
4.4 DevOps in a Multi-Party Setting (Joint Agile Delivery) . 11
4.5 Delivery Pipeline . 11
4.6 Continuous Integration . 12
4.7 Joint Solution Verification Environment . 12
4.8 Continuous and Automated Testing . 12
4.9 Joint Test Design & Execution . 13
4.10 Continuous and Automated Delivery . 13
4.11 Continuous Monitoring . 13
4.12 Processes and Tooling . 13
4.12.1 Process . 13
4.12.2 Tooling . 15
5 Use cases . 15
5.1 Introduction . 15
5.2 Roles . 16
5.3 Initiation of joint pipeline . 16
5.3.1 Concepts . 16
5.3.2 Components . 17
5.3.2.1 DevOps server . 17
5.3.2.2 Data handling component . 17
5.3.3 Preconditions . 18
5.3.4 Interactions . 18
5.3.5 Post conditions . 18
5.4 Delivery . 19
5.4.1 Single VNF Provider to single VNF Operator . 19
5.4.1.1 Roles . 19
5.4.1.2 Preconditions . 19
5.4.1.3 Interactions . 19
5.4.1.4 Post conditions . 19
5.4.2 Multiple VNF Providers to single VNF Operator . 19
5.4.2.1 Roles . 19
5.4.2.2 Preconditions . 19
5.4.2.3 Interactions . 20
5.4.2.4 Post conditions . 20
5.4.3 Single VNF Provider, single VNF Validator to single VNF Operator . 20
5.4.3.0 Introduction . 20
5.4.3.1 Roles . 21
5.4.3.2 Preconditions . 21
5.4.3.3 Interactions . 21
5.4.3.4 Post conditions . 21
ETSI

---------------------- Page: 3 ----------------------
4 ETSI GR NFV-TST 006 V1.1.1 (2020-01)
5.4.4 Multiple VNF Providers, single VNF Validator to single VNF Operator . 21
5.4.4.1 Roles . 21
5.4.4.2 Preconditions . 22
5.4.4.3 Interactions . 22
5.4.4.4 Post conditions . 23
5.5 Error in production . 23
5.6 Tearing down of joint pipeline . 23
5.6.1 Preconditions . 23
5.6.2 Interactions . 24
5.6.3 Post conditions . 24
5.7 Common functionalities . 24
5.7.1 Pull of VNF Package . 24
5.7.1.0 Introduction . 24
5.7.1.1 Roles . 24
5.7.1.2 Preconditions . 24
5.7.1.3 Interactions . 24
5.7.1.4 Post conditions . 25
5.7.2 VNF Package operability validation . 25
5.7.2.0 Introduction . 25
5.7.2.1 Roles . 25
5.7.2.2 Preconditions . 26
5.7.2.3 Interactions . 26
5.7.2.4 Post Conditions . 26
5.7.3 Provide test result . 26
5.7.3.0 Introduction . 26
5.7.3.1 Roles . 26
5.7.3.2 Preconditions . 26
5.7.3.3 Interactions . 26
5.7.3.4 Post Conditions . 27
5.7.4 Operating dashboard . 27
5.7.4.0 Introduction . 27
5.7.4.1 Roles . 27
5.7.4.2 Preconditions . 27
5.7.4.3 Interactions . 27
5.7.4.4 Post Conditions . 27
6 Test steps and approach . 27
6.1 Step 1: Test Definition . 27
6.2 Step 2: Code/VNF Package Shipment . 28
6.3 Step 3: Automated Test Execution . 28
6.4 Step 4: Moving to Production. 29
6.5 Step 5: Collecting operational data . 29
6.6 Test Function Packaging and Shipment . 29
6.7 Installing and Running Test Functions . 29
6.8 Keeping Track of already done tests . 29
6.9 Format of Results in Report . 30
7 Recommendations . 30
7.0 recommendation . 30
7.1 Structure of a VNF Package . 30
7.1.1 VNF Package . 30
7.2 Description of VNF Package content . 30
7.2.1 VNF Package . 30
7.3 VNF Identification . 31
7.3.1 VNF Package . 31
7.4 Test Execution of Test Functions . 31
7.4.1 Modelling testing code as Test VNFs . 31
7.4.1.1 Description . 31
7.4.1.2 Test Network Service . 31
7.4.2 Modelling test as being part of a VNF package . 32
7.4.2.1 Description . 32
7.4.2.2 VNF Package . 32
ETSI

---------------------- Page: 4 ----------------------
5 ETSI GR NFV-TST 006 V1.1.1 (2020-01)
7.4.2.3 VNF. 32
7.5 Acceptance test feedback to VNF provider/developer . 32
7.5.1 Description . 32
7.5.2 VNF . 32
7.5.3 OSS and EM . 33
7.6 Test Specification Languages . 33
7.6.1 Description . 33
7.7 VNFC Software Component Update and Upgrade . 33
7.7.1 Description . 33
7.7.2 VNF Package . 33
7.7.3 NFV-MANO . 34
History . 35


ETSI

---------------------- Page: 5 ----------------------
6 ETSI GR NFV-TST 006 V1.1.1 (2020-01)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Group Report (GR) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI

---------------------- Page: 6 ----------------------
7 ETSI GR NFV-TST 006 V1.1.1 (2020-01)
1 Scope
The present document provides guidance and recommendations on how to leverage DevOps and CI/CD techniques
across the boundary from SW provider to service provider, or any combination of developer, installation and
operational entities. It explores the implications of the processes with regard to the impact of the SW package handoff
between SW provider and service provider, the required functionality in the NFV system, the different deployment and
operational options by:
• Exploring use cases.
• Defining the steps in the process.
• Defining the metrics and measurements.
• Defining the test procedures.
• Defining post handoffs tests.
• Providing recommendations on VNF Package.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GS NFV-REL 006: "Network Functions Virtualisation (NFV); Reliability; Maintaining
Service Availability and Continuity Upon Software Modification".
[i.2] ETSI GS NFV 003: "Network Functions Virtualisation (NFV); Terminology for main concepts in
NFV".
[i.3] ETSI GS NFV-IFA 011 (V3.2.1) (2019-04): "Network Functions Virtualisation (NFV) Release 3;
Management and Orchestration; VNF Descriptor and Packaging Specification".
[i.4] ETSI GS NFV-TST 001 (V1.1.1) (2016-04): "Network Functions Virtualisation (NFV); Pre-
deployment Testing; Report on Validation of NFV Environments and Services".
[i.5] ETSI GR NFV-TST 004 (V1.1.2) (2017-07): "Network Functions Virtualisation (NFV); Testing;
Guidelines for Test Plan on Path Implementation through NFVI".
ETSI

---------------------- Page: 7 ----------------------
8 ETSI GR NFV-TST 006 V1.1.1 (2020-01)
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the terms given in ETSI GS NFV 003 [i.2] and the following apply:
VNF operator: person or company that operates the VNF
VNF validator: person or company that validates the functionality of a VNF
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the abbreviations given in ETSI GS NFV 003 [i.2] and the following apply:
CI Continuous Integration
DA Disciplined Agile
ITSM IT Service Management
JAD Joint Agile Delivery
SAFe Scaled Agile Framework
SDLC Software Development Life Cycle
SRE Site Reliability Engineering
4 Background on Devops and CI/CD
4.1 Introduction
The basic intention in DevOps/Joint Agile Delivery is to incorporate operational and other related aspects of the entire
software lifecycle as early in the development process as practical, and to ensure smooth delivery jointly across
organizations. This requires that many steps in the process are more continuous, incremental, and iterative than in
traditional waterfall models, and higher levels of automation are applied to the delivery pipeline, including testing.
To that end, there are a number of processes and concepts involved; these are described in the following clauses of the
present document.
The context of the present document is the testing elements of DevOps/CI/CD; the present document does not seek to
provide a comprehensive overview of these topics beyond that scope.
4.2 General Backgrounder
DevOps is a combination of practices that embody a culture of end-to-end ownership and shared responsibility across
teams from Development, QA, Security, Operations, and other areas as a single cohesive, service focused team.
DevOps takes much of its heritage from the areas of lean manufacturing and operational management theory. This is
about applying industrial design techniques to the software industry.
Joint Agile Delivery (JAD) is a series of multi-party interlocks to support agile software development and delivery in a
complex carrier environment. It does this through the expected liaison at
...