ETSI GR NFV-TST 006 V1.2.1 (2022-12)

Network Functions Virtualisation (NFV); Testing; Report on CICD and DevOps

RGR/NFV-TST006ed121

General Information

• Status: Not Published
• Technical Committee: NFV TST - Testing, Implementation, and Open Source Working Group
• Current Stage: 12 - Completion
• Due Date: 06-Jan-2023
• Completion Date: 21-Dec-2022

Standards Content (Sample)

GROUP REPORT
Network Functions Virtualisation (NFV);
Testing;
Report on CICD and DevOps
Disclaimer
The present document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

2 ETSI GR NFV-TST 006 V1.2.1 (2022-12)

Reference
RGR/NFV-TST006ed121
Keywords
CI/CD, DevOps, NFV, NFVI, SDN
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - APE 7112B
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° w061004871

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
If you find a security vulnerability in the present document, please report it through our
Coordinated Vulnerability Disclosure Program:
https://www.etsi.org/standards/coordinated-vulnerability-disclosure
Notice of disclaimer & limitation of liability
The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of
experience to understand and interpret its content in accordance with generally accepted engineering or
other professional standard and applicable regulations.
No recommendation as to products and services or vendors is made or should be implied.
No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law
rule and/or regulation and further, no representation or warranty is made of merchantability or fitness
and/or governmental
for any particular purpose or against infringement of intellectual property rights.
In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.

Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not
limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property
rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages
for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use
of or inability to use the software.
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and
microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2022.
All rights reserved.
ETSI
3 ETSI GR NFV-TST 006 V1.2.1 (2022-12)
Contents
Intellectual Property Rights . 6
Foreword . 6
Modal verbs terminology . 6
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definition of terms, symbols and abbreviations . 8
3.1 Terms . 8
3.2 Symbols . 8
3.3 Abbreviations . 8
4 Background on DevOps and CI/CD . 8
4.1 Introduction . 8
4.2 General Backgrounder . 8
4.3 DevOps in an NFV context . 11
4.4 DevOps in a Multi-Party Setting (Joint Agile Delivery) . 11
4.5 Delivery pipeline . 11
4.6 Continuous Integration (CI) . 12
4.7 Joint Solution Verification Environment . 12
4.8 Continuous and Automated Testing . 12
4.9 Joint Test Design & Execution . 13
4.10 Continuous and Automated Delivery . 13
4.11 Continuous Monitoring . 13
4.12 Processes and tooling . 13
4.12.1 Process . 13
4.12.2 Tooling . 15
5 Use cases . 15
5.1 Introduction . 15
5.2 Roles . 16
5.3 Initiation of joint pipeline . 16
5.3.1 Concepts . 16
5.3.2 Components . 17
5.3.2.1 DevOps server . 17
5.3.2.2 Data handling component . 17
5.3.2.3 Test Framework . 18
5.3.3 Preconditions . 19
5.3.4 Interactions . 19
5.3.5 Post conditions . 19
5.4 Delivery . 19
5.4.1 Single VNF Provider to single VNF Operator . 19
5.4.1.1 Roles . 19
5.4.1.2 Preconditions . 20
5.4.1.3 Interactions . 20
5.4.1.4 Post conditions . 20
5.4.2 Multiple VNF Providers to single VNF Operator . 20
5.4.2.1 Roles . 20
5.4.2.2 Preconditions . 20
5.4.2.3 Interactions . 20
5.4.2.4 Post conditions . 21
5.4.3 Single VNF Provider, single VNF Validator to single VNF Operator . 21
5.4.3.0 Introduction . 21
5.4.3.1 Roles . 21
5.4.3.2 Preconditions . 22
5.4.3.3 Interactions . 22
ETSI
4 ETSI GR NFV-TST 006 V1.2.1 (2022-12)
5.4.3.4 Post conditions . 22
5.4.4 Multiple VNF Providers, single VNF Validator to single VNF Operator . 22
5.4.4.1 Roles . 22
5.4.4.2 Preconditions . 22
5.4.4.3 Interactions . 23
5.4.4.4 Post conditions . 24
5.5 Error in production . 24
5.6 Tearing down of joint pipeline . 24
5.6.1 Preconditions . 24
5.6.2 Interactions . 24
5.6.3 Post conditions . 25
5.7 Common functionalities . 25
5.7.1 Pull of VNF Package . 25
5.7.1.0 Introduction . 25
5.7.1.1 Roles . 25
5.7.1.2 Preconditions . 25
5.7.1.3 Interactions . 25
5.7.1.4 Post conditions . 26
5.7.2 VNF Package operability validation . 26
5.7.2.0 Introduction . 26
5.7.2.1 Roles . 26
5.7.2.2 Preconditions . 26
5.7.2.3 Interactions . 27
5.7.2.4 Post Conditions . 27
5.7.3 Provide test result . 27
5.7.3.0 Introduction . 27
5.7.3.1 Roles . 27
5.7.3.2 Preconditions . 27
5.7.3.3 Interactions . 27
5.7.3.4 Post Conditions . 27
5.7.4 Operating feedback . 28
5.7.4.0 Introduction . 28
5.7.4.1 Roles . 28
5.7.4.2 Preconditions . 28
5.7.4.3 Interactions . 28
5.7.4.4 Post Conditions . 28
5.7.5 VNF production deployment . 28
5.7.5.0 Introduction . 28
5.7.5.1 Roles . 29
5.7.5.2 Preconditions . 29
5.7.5.3 Interactions . 29
5.7.5.4 Post Conditions . 29
6 Test steps and approach . 29
6.1 Step 1: Test Definition . 29
6.2 Step 2: Code/VNF Package Shipment . 30
6.3 Step 3: Automated Test Execution . 30
6.4 Step 4: Moving to Production. 30
6.5 Step 5: Collecting operational data . 31
6.6 Test Function Packaging and Shipment . 31
6.7 Installing and Running Test Functions . 31
6.8 Keeping Track of already done tests . 31
6.9 Format of Results in Report . 32
6.10 Information feedback . 32
7 Recommendations . 32
7.0 recommendation . 32
7.1 Structure of a VNF Package . 32
7.1.1 VNF Package . 32
7.2 Description of VNF Package content . 33
7.2.1 VNF Package . 33
7.3 VNF Identification . 33
ETSI
5 ETSI GR NFV-TST 006 V1.2.1 (2022-12)
7.3.1 VNF Package . 33
7.4 Test Execution of Test Functions . 33
7.4.1 Modelling testing code as Test VNFs . 33
7.4.1.1 Description . 33
7.4.1.2 Test Network Service . 34
7.4.2 Modelling test as being part of a VNF package . 34
7.4.2.1 Description . 34
7.4.2.2 VNF Package . 34
7.4.2.3 VNF. 34
7.5 Information feedback to VNF provider/developer . 34
7.5.1 Description . 34
7.5.2 VNF . 35
7.5.3 OSS and EM . 35
7.5.4 NFV-MANO . 35
7.6 Test Specification Languages . 35
7.6.1 Description . 35
7.7 VNFC Software Component Update and Upgrade . 35
7.7.1 Description . 35
7.7.2 VNF Package . 36
7.7.3 NFV-MANO . 36
7.8 VNF Upgrade . 36
7.8.1 Description . 36
7.8.2 NFV-MANO . 37
History . 38

ETSI
6 ETSI GR NFV-TST 006 V1.2.1 (2022-12)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations
pertaining to these essential IPRs, if any, are publicly available for ETSI members and non-members, and can be
found in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to
ETSI in respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the
ETSI Web server (https://ipr.etsi.org/).
Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs,
including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not
referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become,
essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its

Members. 3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and of the 3GPP
Organizational Partners. oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and of the ®
oneM2M Partners. GSM and the GSM logo are trademarks registered and owned by the GSM Association.
Foreword
This Group Report (GR) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI
7 ETSI GR NFV-TST 006 V1.2.1 (2022-12)
1 Scope
The present document provides guidance and recommendations on how to leverage DevOps and CI/CD techniques
across the boundary from SW provider to service provider, or any combination of developer, installation and
operational entities. It explores the implications of the processes with regard to the impact of the SW package handoff
between SW provider and service provider, the required functionality in the NFV system, the different deployment and
operational options by:
• Exploring use cases.
• Defining the steps in the process.
• Defining the metrics and measurements.
• Defining the test procedures.
• Defining post handoffs tests.
• Providing recommendations on VNF Package.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GS NFV-REL 006: "Network Functions Virtualisation (NFV) Release 3; Reliability;
Maintaining Service Availability and Continuity Upon Software Modification".
[i.2] ETSI GR NFV 003: "Network Functions Virtualisation (NFV); Terminology for main concepts in
NFV".
[i.3] ETSI GS NFV-IFA 011 (V3.2.1) (2019-04): "Network Functions Virtualisation (NFV) Release 3;
Management and Orchestration; VNF Descriptor and Packaging Specification".
[i.4] ETSI GS NFV-TST 001 (V1.1.1) (2016-04): "Network Functions Virtualisation (NFV); Pre-
deployment Testing; Report on Validation of NFV Environments and Services".
[i.5] ETSI GR NFV-TST 004 (V1.1.2) (2017-07): "Network Functions Virtualisation (NFV); Testing;
Guidelines for Test Plan on Path Implementation through NFVI".
[i.6] ETSI GS NFV-TST 013: "Network Functions Virtualisation (NFV); Testing; Test Case
Description Template Specification".
[i.7] ETSI GS NFV-SOL 004 (V4.3.1) (2022-07): "Network Functions Virtualisation (NFV) Release 4;
Protocols and Data Models; VNF Package and PNFD Archive specification".
ETSI
8 ETSI GR NFV-TST 006 V1.2.1 (2022-12)
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the terms given in ETSI GR NFV 003 [i.2] and the following apply:
VNF operator: person or company that operates the VNF
VNF validator: person or company that validates the functionality of a VNF
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the abbreviations given in ETSI GR NFV 003 [i.2] and the following apply:
CI Continuous Integration
DA Disciplined Agile
ITSM IT Service Management
JAD Joint Agile Delivery
SAFe Scaled Agile Framework
SDLC Software Development Life Cycle
SRE Site Reliability Engineering
4 Background on DevOps and CI/CD
4.1 Introduction
The basic intention in DevOps/Joint Agile Delivery is to incorporate operational and other related aspects of the entire
software lifecycle as early in the development process as practical, and to ensure smooth delivery jointly across
organizations. This requires that many steps in the process are more continuous, incremental, and iterative than in
traditional waterfall models, and higher levels of automation are applied to the delivery pipeline, including testing.
To that end, there are a number of processes and concepts involved; these are described in the following clauses of the
present document.
The context of the present document is the testing elements of DevOps/CI/CD; the present document does not seek to
provide a comprehensive overview of these topics beyond that scope.
4.2 General Backgrounder
DevOps is a combination of practices that embody a culture of end-to-end ownership and shared responsibility across
teams from Development, QA, Security, Operations, and other areas as a single cohesive, service focused team.
DevOps takes much of its heritage from the areas of lean manufacturing and operational management theory. This is
about applying industrial design techniques to the software industry.
Joint Agile Delivery (JAD) is a series of multi-party interlocks to support agile software development and delivery in a
complex carrier environment. It does this through the expected liaison at key points along the way, such as around
requirements, testing, and acceptance. It leverages CI/CD tools and techniques to ensure robust, repeatable, rapid
delivery of each code iteration. It is compatible with agile software delivery methodologies such as the Scaled Agile
Framework (SAFe), and Disciplined Agile (DA).
ETSI
9 ETSI GR NFV-TST 006 V1.2.1 (2022-12)
As hardware becomes commodity, a new technology base of cloud and virtualisation, microservices, and APIs has
emerged. These technologies have provided an inflection point - an opportunity to re-examine the way in which ICT is
performed and managed; to distil key processes down to their essence, and re-apply those principles in a way that
leverages these technologies to provide a better way to manage quality and change. This new approach has dramatically
altered the risk landscape, and this has an impact on the operating model, if an enterprise is to recognize the benefits of
this new approach.
Some of the most important aspects of this change include:
• Cloud versus bespoke infrastructure solutions. Traditional applications required bespoke infrastructure
solutions. Often the production environment was, due to cost, significantly different from other environments.
This had implications in how it was maintained: managed in-situ across an extended lifespan; all-or-nothing
changes executed in 'quiet' times, often involving service disruption; configuration divergence between
environments from development to test to production having implications on testing and quality assurance.
Cloud, by contrast, has the potential to eliminate all of these concerns.
• Infrastructure as code. The nature of cloud environments is that infrastructure can be created, modified, and
deleted using APIs. This means that it can be created under programmatic control. This implies: repeatability
and consistency, eliminating human error, and elimination of the entropy associated with having to maintain
bespoke, manually configured infrastructure in-situ across an extended life span.
• All environments virtualised. The virtualisation of all environments provides an opportunity to ensure
consistency across environments never previously available (or cost prohibitive). This significantly reduces the
risk associated with change, both due to consistency of the environments, and consistency of the change
process used to promote code between environments.
• Microservices; elimination of heavy integration overheads. A significant source of enterprise risk is the
integration of a large number of changes rolled up into major releases. This leads to extended integration
timeframes, and a lack of direct traceability to the source of errors found during this integration cycle. The
industry trend towards discrete microservices directly addresses this risk, increasing both velocity and quality
simultaneously. The limited scope of each service makes testing more predictable and simpler. The use of
APIs as the entry point protects consumers from internal changes, further simplifying integration testing. Each
service can evolve at its own speed, no longer enforcing large, complex integration cycles.
Against this technology backdrop, DevOps institutes a set of cohesive software delivery practices:
• Continuous Integration. The defining characteristic of DevOps projects is their creation and ubiquitous use
of a delivery pipeline which controls the advancement of release artefacts from idea through to production.
This is not a static pipeline, nor is there only one. The pipeline more closely resembles a factory production
line with many conveyor belts and components being assembled until ultimately brought together to make a
new car. Each of these components have their own pipeline and operate under their own rules and speed.
• Small, discrete changes. A significant source of work and error is the delayed integration of new features and
code changes into the "baseline". Continuous Integration espouses a view that work is continuously checked
into the baseline, to catch any integration related issues early, and ensure that the product is always stable and
passes regression tests. This is then coupled with Continuous Delivery to ensure that a new, clean version of
the product is always shippable. This combination means that there is a confidence in the build and release
process that supports a significantly faster end-to-end release process, further creating a virtuous cycle of
quality and velocity.
• Automated testing. Supporting continuous integration and the rapid promotion of changes through the
delivery pipeline, automation of testing is held out as an essential aspect of DevOps. Not all testing can be
automated, but automation should always be sought to the extent possible.
• High degree of automation. Beyond testing, automation is applied to all steps in the delivery pipeline,
including generation of intermediate artefacts, packaging, environment creation, delivery, deployment and
event response. This significantly increases both the power and authority of the development team.
• Monitoring, metrics, and real world feedback. DevOps is about rapid and continuous feedback. A key focus
is on instrumentation and operational telemetry, and using real world feedback rather than relying on
requirements and test environments.
ETSI
10 ETSI GR NFV-TST 006 V1.2.1 (2022-12)
• Cohesive, cross-functional teams; shared responsibilities. One of the most talked about elements of
DevOps is the integration of previously siloed expertise into a single, cohesive team with shared incentives
and responsibilities.
Beyond these base practices, a number of "industry best" practices are also often cited:
• Version Control and the "Left Edge". Critical to successful DevOps projects is the use of version control
over all source artefacts (not just source code). This applies to less obvious source artefacts, such as visual
workflows and related design artefacts. Developers only have write access to the "left edge" - the source
artefacts. Everything generated from those artefacts is designated an intermediate artefact, that cannot be
directly altered, and is re-generated by committing new source changes. This ensures that version control is
enforced, and that every version of the system, and every element of the system is reproducible.
• Continuous Integration Triggers and Automated Testing. Another aspect of a mature CI delivery pipeline
is the implementation of triggers - that the pipeline CI controller can detect events and automatically take
action, such as continue to promote the object along the delivery pipeline, without human intervention. The
most obvious example of this is where a code check-in causes the automated test suite to be run, in order to
determine if the build is still clean after the most recent code change. CI triggers include source code
repository triggers (code commit), test triggers (success/failure), and human approval triggers (approval to
promote).
• Use of "ephemeral" infrastructure. A major historical source of error was the entropy introduced by the
in-situ maintenance of infrastructure over an extended period. A cloud best practice is to generate new
infrastructure for each new release, build and release the new version of the service onto that infrastructure,
then to cutover/migrate the workloads to this new version in a controlled manner, and eventually destroy the
virtual infrastructure of the previous version. This means that the current and previous version of the service
remain operating in parallel until the migration is complete, and this provides additional mechanisms for
managing risk.
• No rollback, only fail forward. The in-situ nature of traditional large systems meant that change was "all or
nothing", and if a change failed, it would be "rolled back" and the system returned to its previous (pre-change)
state. This technique was always fraught, as any transactions applied from the time of the change would also
need to be rewound and then reapplied, something that was not always possible. It would also often mean
restoring from backups - a procedure often untested except in high risk circumstances such as restoration
during a major event; complications would abound. In mature DevOps environments, a bug is addressed at the
source (left edge) and the tests that allowed the bug to slip through are also addressed. The velocity of the end-
to-end process making it faster and more reliable to push a new version quickly. Where necessary,
functionality might be immediately disabled without any new code change or rollback (see next).
• Decoupling of Deployment from Activation. Advanced DevOps practices include the use of "feature flags"
to decouple deployment from activation. This addresses the most significant risk associated with change - the
all-or-nothing nature of a release. This technique has service code consult a dynamic configuration service to
determine whether a feature should remain dormant, be activated for a particular set of users (e.g. specific
users, specific locations, specific volumes, etc.), or be fully activated for all, and this becomes the basic risk
control mechanism to control the potential impact of a new feature or release.
• Continuous Deployment. With deployment and activation decoupled, delivery pipelines can extend all the
way into production. Continuous Deployment, sometimes called "push on green" is instituted where sufficient
confidence exists in the delivery pipeline, including the automated testing steps.
• Continuous feedback, A/B testing. All of the above features provide an environment of continuous feedback,
where real production data is used to guide development and operations practices and priorities. Feature flags
can be used to present alternate versions of a feature to different users, providing a way to compare their
usefulness and value (A/B testing). This moves software development from the guessing associated with
traditional requirements to the real world feedback of how the system is used.
• Site Reliability Engineering (SRE). A sister movement to DevOps, SRE is the application of IT Service
TM TM TM TM
Management (ITSM) at cloud scale. Pioneered by Google , Amazon , Netflix and other Silicon Valley
cloud companies, SRE is the application of automation to ICT operations, and the evolution of the IT
operations function.
ETSI
11 ETSI GR NFV-TST 006 V1.2.1 (2022-12)
4.3 DevOps in an NFV context
In the context of Network Function Virtualisation, all of the above benefits can be brought into play. Smaller,
incremental releases of VNFs and VNFCs can be rapidly promoted through a continuous integration/delivery pipeline
as described in the present document. The use of discrete function
...