ETSI TR 103 761 V1.1.1 (2022-05)

Core Network and Interoperability Testing (INT); Methodologies for E2E Testing & Validation of Vertical Applications over 5G & Beyond networks

DTR/INT-00170

General Information

• Status: Not Published
• Technical Committee: INT - Core Network and Interoperability Testing
• Current Stage: 12 - Completion
• Due Date: 06-Jul-2022
• Completion Date: 06-May-2022

Standards Content (Sample)

TECHNICAL REPORT
Core Network and Interoperability Testing (INT);
Methodologies for E2E Testing & Validation of
Vertical Applications over 5G & Beyond networks

2 ETSI TR 103 761 V1.1.1 (2022-05)

Reference
DTR/INT-00170
Keywords
5G, core network, methodology
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - APE 7112B
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° w061004871

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Notice of disclaimer & limitation of liability
The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of
experience to understand and interpret its content in accordance with generally accepted engineering or
other professional standard and applicable regulations.
No recommendation as to products and services or vendors is made or should be implied.
No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law
and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness
for any particular purpose or against infringement of intellectual property rights.
In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.

Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not
limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property
rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages
for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use
of or inability to use the software.
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and
microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2022.
All rights reserved.
ETSI
3 ETSI TR 103 761 V1.1.1 (2022-05)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Executive summary . 5
Introduction . 6
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definition of terms, symbols and abbreviations . 12
3.1 Terms . 12
3.2 Symbols . 13
3.3 Abbreviations . 13
4 Methodologies for Testing and Validation of Vertical Applications over 5G & Beyond Networks . 16
4.1 Motivation and expectations. 16
4.1.1 Stakeholders . 16
4.1.2 Expectations . 17
4.1.2.1 Application-network interoperability verification . 17
4.1.2.2 Application end-to-end performance assurance and characterization . 17
4.1.2.3 Network technology assessment and selection for serving the application under test . 18
4.1.2.4 Application deployment model evaluation and optimization . 18
4.1.3 Key Variables: Inputs and Outputs . 18
4.1.4 The Validation Cycle . 18
4.1.5 The Validation Framework . 19
4.2 State of the Art survey . 19
4.2.1 Standards of relevance . 19
4.2.1.1 3GPP . 19
4.2.1.2 ITU . 20
4.2.1.3 ETSI . 20
4.2.2 Industry Alliance in the Vertical ecosystem: 5G ACIA . 22
4.2.3 The European Commission (EC) 5G ICT projects as state of the art . 22
4.2.3.1 Introduction to 5G PPP program . 22
4.2.3.2 5G PPP TMV . 23
4.2.3.2.1 Testing Methodologies and Testing as a Service . 23
4.2.3.2.2 Essential KPIs for Service Validation . 24
4.2.3.3 5G PPP validation platform solutions: 5G-EVE, 5GENESIS and 5G-VINNI . 24
4.2.3.4 Future Internet Research and Experimentation (FIRE) . 24
4.2.4 Assessment of the state of the art solutions . 25
4.2.4.1 Comparison of 5G PPP validation solutions . 25
4.2.4.2 Top-5 key features in state-of-the-art validation platforms . 28
4.2.4.3 Top-5 potential enhancements in state-of-the-art validation platforms . 29
4.2.4.4 Assessment conclusions . 30
4.3 Recommendations for the Validation Framework . 30
4.3.1 Introduction. 30
4.3.2 Recommendations . 31
4.3.2.1 Deployment and reference model . 31
4.3.2.2 Capabilities. 32
4.3.2.2.1 5G Capabilities & Enablers . 32
4.3.2.2.2 Testing and Validation Environment . 35
4.3.2.2.3 Processes . 40
4.3.2.2.4 KPI Mechanisms . 47
4.3.2.2.5 Vertical Applications Design . 56
ETSI
4 ETSI TR 103 761 V1.1.1 (2022-05)
5 Conclusion . 57
Annex A: EC 5G PPP platform solutions . 59
A.1 5G PPP 5G EVE Platform . 59
A.2 KPI collection and validation in 5G EVE Platform . 60
A.3 5G VINNI Platform . 64
A.4 The 5G-VINNI Testing-as-a-Service system . 65
A.5 5GENESIS Platform. 67
Annex B: EC FIRE programs . 69
Annex C: SLA requirements on "Predictive QoS" for Automotive / V2X applications and on
"Time Sensitive KPIs"" for Industry 4.0 . 71
C.0 Background . 71
C.1 Use Cases of Federated AMC Knowledge Planes (Inter-Domain and Inter Operator option) for
E2E Network Slice Autonomic Service Assurance . 71
C.2 Description of possible options . 72
C.2.1 Option-A (Horizontal Federation) in Inter-Domain model (a single Organization) . 72
C.2.2 Option-B (Hierarchical /Vertical Federation) in Inter-Domain model (a single Organization) . 73
C.2.3 Option-B (Hierarchical /Vertical Federation) in Inter-Operator model (Multi Organization) . 75
History . 76

ETSI
5 ETSI TR 103 761 V1.1.1 (2022-05)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations
pertaining to these essential IPRs, if any, are publicly available for ETSI members and non-members, and can be
found in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to
ETSI in respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the
ETSI Web server (https://ipr.etsi.org/).
Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs,
including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not
referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become,
essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its

Members. 3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and of the 3GPP
Organizational Partners. oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and of the ®
oneM2M Partners. GSM and the GSM logo are trademarks registered and owned by the GSM Association.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Core Network and Interoperability
Testing (INT).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Executive summary
The purpose of the present document is to provide recommendations on methodologies for end-to-end testing and
validation of vertical applications over 5G and beyond networks. The present document includes recommendations
covering the most aspects of a B5G-app validation framework by providing recommendation on B5G capabilities and
enablers, on the testing and validation environment, on involved processes, on the relevant KPI mechanisms and,
finally, on the design of vertical applications under test. Such recommendations can be equally applicable to a wide
range of industry verticals, application cases and beyond 5G scenarios.
The main value of such end-to-end testing and validation activity is the fact that the vertical application provider can
experiment with the 5G and beyond network in order to make business decisions previously to going into
commercialization. In this context it is assumed that the subject under test is the application and that the 5G and beyond
network setup (as well as the configurations considered in such experimentation) have already been tested and qualified
both from functionality and performance perspective.
ETSI
6 ETSI TR 103 761 V1.1.1 (2022-05)
To that end, the present document provides a survey and review of the existing methodologies for testing and validating
vertical applications, leveraging on the experiences gain through several innovation projects. This exercise permits the
identification of existing gaps in such methodologies, proposing solutions to cover them. The present document
describes Processes, Mechanisms and Strategies involved in the testing and validation of innovative vertical
applications enabled by 5G and Beyond networks. The work does not consider any assumption on the specific business
or nature of the vertical domain, with the intention to identify common methodologies applicable to as wide a range of
vertical domains as possible.
Introduction
Most vertical industries are transforming their processes and innovating their business model, and for that purpose they
are actively exploring and adopting a wide range of new technologies. In particular, the adoption of 5G for overcoming
limitations and challenges of connectivity and flexibility of other technologies is regarded instrumental for their
success. The new 5G landscape of architectures, evolving features and superior performance levels enables possibilities
for vertical industries in its digital transformation, and therefore 5G has become a subject of priority focus all along
their innovation life-cycle for new applications and solutions, from business opportunity identification to new
application's design, solution integration and technical and business validation.
Actually, from early stages of 5G standardization to its regulation, and first commercial deployments, a number of
vertical industries have engaged not only on proactive surveillance of 5G technology but also in tight collaborations
with Communication Service Providers (CSPs), Telecommunications Equipment Vendors, Academic and Research
Institutions and start-ups. That has been a major factor in the steering and shaping of new innovation ecosystems around
5G all around the world, being a remarkable example the one boosted in Europe by the 5G Infrastructure Public Private
Partnership (5G PPP) which is a joint initiative between the European Commission and European ICT industry.
Virtually all initiatives and projects have been promoted by the 5G PPP. On the one hand, 5G PPP has analysed
transformation opportunities of major players of vertical industry sectors, with special attention to end-to-end
application requirements and, in turn, their expectations on connectivity and flexibility of the underlying 5G network.
On the other hand, 5G PPP has studied and validated the feasibility of 5G architectures and solutions for fulfilling those
expectations.
This type of prior-to-commercialization critical validation activities, and that of their implicit challenges motivated, in
2018, the substantiation of large European infrastructures (namely 5G EVE, 5G-VINNI and 5GENESIS projects) for
effectively and efficiently host and run the increasing number of 5G-ready application validation activities in the 5G
PPP ecosystem. Furthermore, in 2019, the experience and learnings in validation activities translated into the creation of
a special 5G PPP workgroup devoted to the applied science of Testing, Measurement and Validation (TMV). The
workgroup has been collecting and analysing rich and varied information, from a broad set of projects, on their
challenges, approaches, methodologies and tools producing guidelines and recommendations for piloting, adoption,
design and execution of validation activities.
Such experience can serve as a good basis for sharing and applying beyond the 5G PPP ecosystem. And given the pace
of evolution of 5G towards B5G networks, combined with the intense innovation in vertical applications, a further and
careful look into the upcoming challenges for validating applications over B5G networks, seems to be well justified too.
So, the ambition of the present document is to leverage the first-hand experience and learnings in 5G PPP, and to assess
and provide a basic set of recommendations for crafting effective capabilities, processes and mechanisms for validating
vertical applications over 5G and beyond networks.
Despite the reference work being based on initiatives triggered at European level, the outcomes of the present document
intend to be globally applicable, without limitation to specific geographic conditions or circumstances. The focus of
analysis also sees to that the expectations from the relevant stakeholders involved in the innovation and validation cycle
of innovative vertical applications relying on 5G and beyond networks are specified, analysed and supported by the
proposed recommendations.
ETSI
7 ETSI TR 103 761 V1.1.1 (2022-05)
1 Scope
The purpose of the present document is to provide recommendations on methodologies for end-to-end testing and
validation of vertical applications over 5G and beyond networks. Such recommendations can be equally applicable to a
wide range of industry verticals, application cases and beyond 5G scenarios.
By applying such standard end to end testing and validation methodologies, the vertical application provider would be
able to experiment with the 5G and beyond network in a systematic and consistent manner and make informed business
decisions upon about further development and commercialization of the features of its application under that rely on 5G
technologies and beyond. In this context it is assumed that the subject under test is the application and that the 5G and
beyond network set up and configurations considered in such experimentation have already been tested and qualified
both from functionality and performance perspective.
Therefore, general (application-independent) testing and characterization of 5G network setups is not in the scope of
analysis of the present document. Actually, special emphasis on making clear distinction between network testing and
application validation concerns and distinct challenges is secured along the analysis since the processes, mechanisms,
tools and strategies typically used for network testing can prove inadequate or misleading for vertical application
validation purposes.
The present document provides a survey and review of the existing methodologies for testing and validating vertical
applications, to identify existing gaps in such methodologies and propose solutions to cover them. The proposed
methodology describes Capabilities, Processes and Mechanisms involved in the testing and validation of innovative
vertical applications enabled by 5G and Beyond networks.
No assumptions are made on the specific business or nature of the vertical domain, with the intention to identify
common methodologies applicable to as wide a range of vertical domains as possible. The analysis of specific
methodologies for applicability limited to specific vertical domains is beyond the scope of the present document.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] 5G PPP Test, Measurement, and KPIs Validation (TMV) Working Group white paper: "Validating
5G Technology Performance Assessing 5G architecture and Application Scenarios", June 2019.
NOTE: Available at https://5g-ppp.eu/wp-content/uploads/2019/06/TMV-White-Paper-V1.1-25062019.pdf.
[i.2] ETSI TS 138 521-3 (V15.4.1): "5G; NR; User Equipment (UE) conformance specification; Radio
transmission and reception; Part 3: Range 1 and Range 2 Interworking operation with other radios
(3GPP TS 38.521-3 version 15.4.1 Release 15)".
[i.3] ETSI TS 138 521-2 (V15.4.0): "5G; NR; User Equipment (UE) conformance specification; Radio
transmission and reception; Part 2: Range 2 standalone (3GPP TS 38.521-2 version 15.4.0
Release 15)".
ETSI
8 ETSI TR 103 761 V1.1.1 (2022-05)
[i.4] ETSI TS 138 521-1 (V15.3.0): "5G; NR; User Equipment (UE) conformance specification; Radio
transmission and reception; Part 1: Range 1 standalone (3GPP TS 38.521-1 version 15.3.0
Release 15)".
[i.5] ETSI TS 138 521-4 (V15.2.0): "5G; NR; User Equipment (UE) conformance specification; Radio
transmission and reception; Part 4: Performance (3GPP TS 38.521-4 version 15.2.0 Release 15)".
[i.6] ETSI TS 138 533 (V15.2.0): "5G; NR; User Equipment (UE) conformance specification; Radio
Resource Management (RRM) (3GPP TS 38.533 version 15.2.0 Release 15)".
[i.7] ETSI TR 137 901 (V15.1.0): "Universal Mobile Telecommunications System (UMTS); LTE; User
Equipment (UE) application layer data throughput performance (3GPP TR 37.901 version 15.1.0
Release 15)".
[i.8] ETSI TR 137 901-5 (V16.3.0): "5G; Study on 5G NR User Equipment (UE) application layer data
throughput performance (3GPP TR 37.901-5 version 16.3.0 Release 16)".
[i.9] ETSI TS 128 554 (V15.5.0): "5G; Management and orchestration; 5G end to end Key Performance
Indicators (KPI) (3GPP TS 28.554 version 15.5.0 Release 15)".
[i.10] ETSI TS 128 552 (V15.6.0): "5G; Management and orchestration; 5G performance measurements
(3GPP TS 28.552 version 15.6.0 Release 15)".
[i.11] 3GPP TS 28.553 (V0.4.0) (July 2018): "Management and orchestration of networks and network
slicing; 5G Core Network (5GC) performance measurements and assurance data".
[i.12] ETSI TS 132 425 (V15.3.0): "LTE; Telecommunication management; Performance Management
(PM); Performance measurements Evolved Universal Terrestrial Radio Access Network (E-
UTRAN) (3GPP TS 32.425 version 15.3.0 Release 15)".
[i.13] ETSI TS 132 450 (V15.1.0): "Universal Mobile Telecommunications System (UMTS); LTE;
Telecommunication management; Key Performance Indicators (KPI) for Evolved Universal
Terrestrial Radio Access Network (E-UTRAN): Definitions (3GPP TS 32.450 version 15.1.0
Release 15)".
[i.14] Recommendation ITU-T Q.API4TB: "Open APIs for interoperable testbed federations", work in
progress, 2021.
[i.15] Recommendation ITU-T Q.3960: "Framework of Internet related performance measurements".
[i.16] Supplement 71 to ITU-T Q-series of Recommendations: "Testing methodologies of Internet
related performance measurements including e2e bit rate within the fixed and mobile operator's
networks".
[i.17] ETSI TS 103 222-4 (V1.1.1): "Speech and multimedia Transmission Quality (STQ); Reference
benchmarking, background traffic profiles and KPIs; Part 4: Reference benchmarking for IPTV,
Web TV and RCS-e Video Share".
NOTE: Available at
https://www.etsi.org/deliver/etsi_ts/103200_103299/10322204/01.01.01_60/ts_10322204v010101p.pdf.
[i.18] ETSI TS 103 222-3 (V1.1.1): "Speech and multimedia Transmission Quality (STQ); Reference
benchmarking, background traffic profiles and KPIs; Part 3: Reference benchmarking, background
traffic profiles and KPIs for UMTS and VoLTE".
NOTE: Available at
https://www.etsi.org/deliver/etsi_ts/103200_103299/10322203/01.01.01_60/ts_10322203v010101p.pdf.
[i.19] ETSI TS 103 222-2 (V1.1.1): "Speech and multimedia Transmission Quality (STQ); Reference
benchmarking, background traffic profiles and KPIs; Part 2: Reference benchmarking and KPIs for
High speed internet".
NOTE: Available at
https://www.etsi.org/deliver/etsi_ts/103200_103299/10322202/01.01.01_60/ts_10322202v010101p.pdf.
ETSI
9 ETSI TR 103 761 V1.1.1 (2022-05)
[i.20] ETSI TS 103 195-2 (V1.1.1): "Autonomic network engineering for the self-managing Future
Internet (AFI); Generic Autonomic Network Architecture; Part 2: An Architectural Reference
Model for Autonomic Networking, Cognitive Networking and Self-Management".
NOTE: Available at
https://www.etsi.org/deliver/etsi_ts/103100_103199/10319502/01.01.01_60/ts_10319502v010101p.pdf.
[i.21] White Paper No.5 of the ETSI 5G PoC: "Artificial Intelligence (AI) in Test Systems, Testing AI
Models and ETSI GANA Model's Cognitive Decision Elements (DEs) via a Generic Test
Framework for Testing GANA Multi-Layer Autonomics & their AI Algorithms for Closed-Loop
Network Automation".
NOTE: Available at
https://intwiki.etsi.org/images/archive/20200527152913%21ETSI_5G_PoC_White_Paper_No_5.pdf.
[i.22] ETSI TC INT/AFI WG PoC (Proof-Of-Concept) Program on 5G Network Slices Creation,
Autonomic & Cognitive Management & End-to-End (E2E) Orchestration; with Closed-Loop
(Autonomic) Service Assurance of 5G Slices.
NOTE: Available at https://intwiki.etsi.org/index.php?title=Accepted_PoC_proposals.
[i.23] ETSI TR 103 748: "INT Artificial Intelligence (AI) in Test Systems and Testing AI models; Use
and benefits of AI technologies in Testing,".
NOTE: Available at https://portal.etsi.org/webapp/WorkProgram/Report_WorkItem.asp?WKI_ID=59455.
[i.24] ETSI TR 103 749: "INT Artificial Intelligence (AI) in Test Systems and Testing AI models;
Testing of AI with definition of quality metrics".
NOTE: Available at https://portal.etsi.org/webapp/WorkProgram/Report_WorkItem.asp?WKI_ID=59456.
[i.25] ETSI TR 103 763: "Core Network and Interoperability Testing (INT); Description of Test
Requirements and Approach for E2E Federated Testbeds".
NOTE: Available at https://portal.etsi.org/webapp/WorkProgram/Report_WorkItem.asp?WKI_ID=59577.
[i.26] NGMN Alliance: "5G End-to-End Architecture Framework", v3.0.8.
NOTE: Available at https://www.ngmn.org/publications/5g-end-to-end-architecture-framework-v3-0-8.html.
[i.27] 5G-ACIA: "5G for Connected Industries and Automation", Second Edition, White Paper, 5G
ACIA, February 2019.
[i.28] 5G-ACIA: "5G for Automation in Industry - Primary use cases, functions and service
requirements", White Paper, 5G ACIA, March 2019.
[i.29] 5G-ACIA: "5G Non-Public Networks for Industrial Scenarios", White Paper, 5G ACIA,
July 2019.
[i.30] 5G-ACIA: "Key 5G Use Cases and Requirements - From the Viewpoint of Operational
Technology Providers", White Paper, 5G ACIA, May 2020.
[i.31] 5G-ACIA: "Integration of 5G with Time-Sensitive Networking for Industrial Communications",
White Paper, 5G-ACIA, January 2021.
[i.32] 5G PPP Phase 1 projects.
NOTE: Available at https://5g-ppp.eu/5g-ppp-phase-1-projects/.
[i.33] 5G PPP Phase 2 projects.
NOTE: Available at https://5g-ppp.eu/5g-ppp-phase-2-projects/.
[i.34] 5G PPP Phase 3 projects.
NOTE: Available at https://5g-ppp.eu/5g-ppp-phase-3-projects/.
ETSI
10 ETSI TR 103 761 V1.1.1 (2022-05)
[i.35] 5G-EVE Project.
NOTE: Available at https://www.5g-eve.eu/.
[i.36] 5GENESIS Project.
NOTE: Available at https://5genesis.eu/.
[i.37] 5G-VINNI Project.
NOTE: Available at https://www.5g-vinni.eu/.
[i.38] OSM.
NOTE: Available at https://osm.etsi.org/.
[i.39] OPNFV.
NOTE: Available at https://www.opnfv.org/.
[i.40] ONAP.
NOTE: Available at https://www.onap.org/.
[i.41] ETSI ZSM.
NOTE: Available at https://www.etsi.org/committee/zsm.
[i.42] 5GPP Test, Measurement, and KPIs Validation (TMV) Working Group white paper: "Service
performance measurement methods over 5G experimental networks", May 2021.
NOTE: Available at https://5g-ppp.eu/wp-content/uploads/2021/06/Service-performance-measurement-methods-
over-5G-experimental-networks_08052021-Final.pdf.
[i.43] ETSI TS 138 101-1 (V15.13.0): "5G; NR; User Equipment (UE) radio transmission and reception;
Part 1: Range 1 Standalone (3GPP TS 38.101-1 version 15.13.0 Release 15)".
[i.44] ETSI TS 138 101-2 (V15.13.0): "5G; NR; User Equipment (UE) radio transmission and reception;
Part 2: Range 2 Standalone (3GPP TS 38.101-2 version 15.13.0 Release 15)".
[i.45] ETSI TS 138 211 (V15.9.0): "5G; NR; Physical channels and modulation (3GPP TS 38.211
version 15.9.0 Release 15)".
[i.46] ETSI TS 128 530 (V15.1.0): "5G; Management and orchestration; Concepts, use cases and
requirements (3GPP TS 28.530 version 15.1.0 Release 15)".
[i.47] Recommendation ITU-T Y.3100 (09/2017): "Terms and definitions for IMT-2020 network".
[i.48] ETSI TS 123 501 (V15.4.0): "5G; System Architecture for the 5G System (3GPP TS 23.501
version 15.4.0 Release 15)".
[i.49] ETSI MEC.
NOTE: Available at https://www.etsi.org/committee/mec.
[i.50] Linux Foundation Akraino.
NOTE: Available at https://www.lfedge.org/projects/akraino/.
[i.51] Openess.
NOTE: Available at https://www.openness.org.
[i.52] ETSI NFV.
NOTE: Available at https://www.etsi.org/committee/nfv.
ETSI
11 ETSI TR 103 761 V1.1.1 (2022-05)
[i.53] iPerf.
NOTE: Available at https://iperf.fr/.
[i.54] ETSI GS NFV-SOL 005 (V2.7.1): "Network Functions Virtualisation (NFV) Release 2; Protocols
and Data Models; RESTful protocols specification for the Os-Ma-nfvo Reference Point".
[i.55] ETSI TS 129 522 (V15.8.0): "5G; 5G System; Network Exposure Function Northbound APIs;
Stage 3 (3GPP TS 29.522 version 15.8.0 Release 15)".
[i.56] ETSI TS 129 222 (V15.9.0): "5G; LTE; Common API Framework for 3GPP Northbound APIs
(3GPP TS 29.222 version 15.9.0 Release 15)".
[i.57] 3GPP TR 23.758 (V17.0.0): "Study on application architecture for enabling Edge Applications".
[i.58] ETSI GS MEC 012 (V2.1.1): "Multi-access Edge Computing (MEC); Radio Network Information
API".
[i.59] ETSI GS MEC-DEC 032-3 (V2.1.1): "Multi-access Edge Computing (MEC); API Conformance
Test Specification; Part 3: Abstract Test Suite (ATS)".
[i.60] Report ITU-R M.2410-0 (11/2017): "Minimum requirements related to technical performance for
IMT-2020 radio interface(s)".
[i.61] 5G-VINNI Deliverable D4.1: "Initial report on test-plan creation and methodology, and
development of test orchestration framework", July 2019.
NOTE: Available at https://zenodo.org/record/3345626.
[i.62] ETSI TR 103 747: "Core Network and Interoperability Testing (INT/ WG AFI); Federated GANA
Knowledge Planes (KPs) for Multi-Domain Autonomic Management & Control (AMC) of Slices
in the NGMN(R) 5G End-to-End Architecture Framework".
[i.63] Recommendation ITU-T Y.3324: "Requirements and architectural framework for autonomic
management and control of IMT-2020 networks".
[i.64] ETSI TR 103 473: "Evolution of management towards Autonomic Future Internet (AFI);
Autonomicity and Self-Management in the Broadband Forum (BBF) Architectures".
[i.65] ETSI TR 103 404: "Network Technologies (NTECH); Autonomic network engineering for the
self-managing Future Internet (AFI); Autonomicity and Self-Management in the Backhaul and
Core network parts of the 3GPP Architecture".
[i.66] ETSI TS 132 404: "Digital cellular telecommunications system (Phase 2+) (GSM); Universal
Mobile Telecommunications System (UMTS); LTE; Telecommunication management;
Performance Management (PM); Performance measurements; Definitions and template (3GPP
TS 32.404)".
[i.67] 3GPP TR 22.804: "Study on Communication for Automation in Vertical domains (CAV)".
[i.68] ETSI TS 122 261: "5G; Service requirements for the 5G system (3GPP TS 22.261)".
[i.69] ETSI TS 122 104: "5G; Service requirements for cyber-physical control applications in vertical
domains (3GPP TS 22.104)".
[i.70] 3GPP TR 22.830: "Study on business role models for network slicing".
[i.71] 3GPP TS 28.531 (V16.0.0): "Management and orchestration; Provisioning".
[i.72] White paper 5G PPP: "5G network support of vertical industries in the 5G Public-Private
Partnership ecosystem".
NOTE: Available at https://5g-ppp.eu/wp-content/uploads/2020/02/Vertical-industries-in-the-5G-PPP.pdf.
ETSI
12 ETSI TR 103 761 V1.1.1 (2022-05)
[i.73] ETSI GANA Model in 5G Network Slicing PoC White Paper #4: "ETSI GANA as Multi-Layer
Artificial Intelligence (AI) Framework for Implementing AI Models for Autonomic Management
& Control (AMC) of Networks and Services; and Intent-Based Networking (IBN) via GANA
Knowledge Planes (KPs)".
NOTE: Available at https://intwiki.etsi.org/index.php?title=Accepted_PoC_proposals.
[i.74] TMForum IG1127: "End-to-end Virtualization Management: Impact on E2E Service Assurance
and SLA Management for Hybrid Networks".
[i.75] ETSI GANA Model in 5G Network Slicing PoC White Paper #3: "Programmable Traffic
Monitoring Fabrics that enable On-Demand Monitoring and Feeding of Knowledge into the ETSI
GANA Knowledge Plane for Autonomic Service Assurance of 5G Network Slices; and
Orchestrated Service Monitoring in NFV/Clouds".
NOTE: Available at https://intwiki.etsi.org/index.php?title=Accepted_PoC_proposals.
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
5G PPP Facility: each one of the testing sites which form an ICT-17 5G validation platform. Thus, each platform has
several facilities in different geographical locations
5G Platform: ICT-17 platforms funded by EU Commission: 5GEVE, 5GVINNI and 5Genesis
5GS observation points: observation points located on interfaces within the 5G System, including the 5G Radio, Edge,
Transport and 5G Packet Core
APP E2E observation points: observation points located on the hardware and software application or services that the
vertical controls or owns
B5G Network: beyond 5G Networks are networks built with technology that is specified by future releases of 3GPP
after release 17, and are planned to be introduced starting 2025
Communication Service Provider (CSP): company that offers communication services, typically, Network Operators
offering Public Mobile services communications
experiment: running of a specific process that includes End user devices, 5G Network components and Vertical
application to discover KPI values that are not know in advance
experiment blueprint: set of composed actions including end user devices, network components, vertical application,
test cases, measurements, and KPIs that can be introduced in a 5G Facility to characterize the behaviour of the system
under specific configuration
Management and Orchestration (MANO): framework developed by a working group of the same name within the
European Telecommunications Standards Institute (ETSI) Industry Specification Group for NFV (ETSI ISG NFV)
NOTE: It is the ETSI-defined framework for the management and orchestration of all resources in a virtualized
data centre including computer, networking, storage, and Virtual Machine (VM) and Container resources.
(Network) Monitoring: monitoring is a computer network's systematic effort to detect slow or failing network
components, such as overloaded or crashed/frozen servers, failing routers, failed switches or other problematic devices
NOTE: In the event of a network failure or similar outage, the network monitoring system alerts the network
administrator. Network monitoring is a subset of network management.
performance: in the context of Networking, analysis and review of collective network statistics, to define the quality of
services offered by the system considering end to end interactions between end user devices and vertical applications
ETSI
13 ETSI TR 103 761 V1.1.1 (2022-05)
subject under test: artefact that is being evaluated for testing purposes
NOTE: Typically, in the context of 5G Platforms, the subject under test matches with Vertical Applications.
test: process of validating either a functional or non-functional behavior of a system (e.g. device, software component,
etc.)
NOTE: In the context of the present document, a Test consist in running a specific process that includes End user
devices, 5G Network components and Vertical Application to obtain KPI values and verify that obtained
values fit in predefined thresholds.
T&M methodologies: methods, rules and processes required to test and measure results of these tests
Test, Measurement, and KPIs Validation (TMV) Working Group: one of the working groups in 5G PPP whose
objective is to bring together the projects within the 5G PPP that have common interest in the development of Test and
Measurement methods, test cases, procedures and KPI validation
Test as a Service (TaaS): automation and interfacing layer that allows to connect all the Test & Measurement tools
needed for validating and verifying a system, from the individual components up to the E2E service
NOTE: It speeds up repeating tests and validating proper behaviour of a system after introducing changes.
testing: process of evaluating a system or its component(s) with the intent to find whether it satisfies the specified
requirements
validation cycle: collaborative process between Vertical industries and Communication Service Providers sharing
objectives, timelines, outcomes and learnings to guarantee proper integration between Vertical applications and
Communications networks
vertical application: software program that performs specific data processing related to specific domain. Examples of
vertical are Factory of the Future (FoF), Unmanned Aerial Systems (UAS), Vehicle to Infrastructure Communications
(V2X) and Edge Applications (EDGEAPP)
vertical KPI model: relationship between service KPIs as defined by the vertical and network KPIs enforced by the
provider
NOTE: It refers to a model which represents the influence of network service KPIs on the Vertical-level KPIs. It
can be presented in a tabular structure (mapping) or in a more complex form.
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
5G PPP 5G Infrastructure Public Private Partnership
5G-RAN 5G-Radio Access Network
5GS 5G System
AFI Autonomic Management and Control Intelligence for Self-Managed Fixed & Mobile Integrated
Networks
AI Artificial Intelligence
AIM Automated Intelligent Management
AMC Autonomic Management and Control
AMF Access and Mobility Management Function
AN Access Network
API Application Programming Interface
APP Application
B5G Beyond 5G
BBF Broadband Forum
CC Component Carriers
CI/CD Continuous Integration/Continuous Deployment
ETSI
14 ETSI TR 103 761 V1.1.1 (2022-05)
CP Cyclic Prefix
CPU Central Processing Unit
CSC Communication Service Customer
CSP Communication Service Provider
CSV Comma Separated Value
DL Downlink
DMRS Demodulation Reference Signal
DTR Draft Technical Report
E2E End-to-End
EEM Experiment Execution Manager
ELCM Experiment LifeCycle Manager
eMBB enhanced Mobile Broadband
EU European Union
FDD Frequen
...