ETSI GR ENI 010 V1.1.1 (2021-03)

Experiential Networked Intelligence (ENI); Evaluation of categories for AI application to Networks

DGR/ENI-0020_Eva_Cat_AI_Ap_Net

General Information

• Status: Not Published
• Technical Committee: ENI - Experiential Networked Intelligence
• Current Stage: 12 - Completion
• Due Date: 19-Feb-2021
• Completion Date: 05-Mar-2021

Standards Content (Sample)

ETSI GR ENI 010 V1.1.1 (2021-03)






GROUP REPORT
Experiential Networked Intelligence (ENI);
Evaluation of categories for AI application to Networks
Disclaimer
The present document has been produced and approved by the Experiential Networked Intelligence (ENI) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

---------------------- Page: 1 ----------------------
2 ETSI GR ENI 010 V1.1.1 (2021-03)



Reference
DGR/ENI-0020_Eva_Cat_AI_Ap_Net
Keywords
artificial intelligence, categorization, network
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2021.
All rights reserved.

DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.

3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI GR ENI 010 V1.1.1 (2021-03)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
1 Scope . 5
2 References . 5
2.1 Normative references . 5
2.2 Informative references . 5
3 Definition of terms, symbols and abbreviations . 5
3.1 Terms . 5
3.2 Symbols . 6
3.3 Abbreviations . 6
4 Introduction . 6
4.1 Background on categories for AI application to networks . 6
4.2 The motivation for evaluating categories of AI application to network . 11
4.3 Responsibility Index in Autonomous Network . 11
5 Evaluation criteria of categories for AI application to Network . 14
5.1 Framework of quantitative evaluation process . 14
5.2 Scoring principles and specification of the single scenario . 16
5.2.1 Scoring principles and specification . 16
5.2.2 Weights determined by Analytic Hierarchy Process . 17
5.2.3 Recommended values of KPI for each intelligent level in some scenarios . 19
5.2.4 Presentation of evaluation result . 21
5.3 Scoring principles and specification of the part of network lifecycle . 21
5.3.1 Explorations on evaluation of network deployment autonomicity categories . 21
5.3.2 Explorations on evaluation of evaluation of the entire network autonomicity categories . 24
6 Application of quantitative evaluation criteria . 29
6.1 Transport Network: 5G and DCN examples . 29
6.1.1 General categorization criteria . 29
6.1.2 5G transport example . 32
6.1.3 DCN transport example . 34
7 Requirements of network infrastructure in different Categories . 36
7.1 Knowledge Base . 36
7.2 Tool for decision delegation according to Time Dimension . 36
8 Conclusions . 37
Annex A: Change History . 39
History . 40


ETSI

---------------------- Page: 3 ----------------------
4 ETSI GR ENI 010 V1.1.1 (2021-03)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Group Report (GR) has been produced by ETSI Industry Specification Group (ISG) Experiential Networked
Intelligence (ENI).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI

---------------------- Page: 4 ----------------------
5 ETSI GR ENI 010 V1.1.1 (2021-03)
1 Scope
The purpose of present document is to define quantitative evaluation criteria of network autonomicity categories, which
are defined in the published ETSI GR ENI 007 [i.2].
The present document is composed of three components:
1) to further define the categories and quantitative factors determining the network autonomicity categories;
2) to define a framework of quantitative evaluation process and a scoring criteria;
3) to describe several scenario examples of quantitative evaluation criteria.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GR ENI 004 (V2.1.1): "Experiential Networked Intelligence (ENI); Terminology for Main
Concepts in ENI".
[i.2] ETSI GR ENI 007 (V1.1.1): "Experiential Networked Intelligence (ENI); ENI Definition of
Categories for AI Application to Networks".
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
autonomous networks: set of self-governing programmable and explainable systems that seamlessly deliver secure,
context-aware, business-driven services that are created and maintained using model-driven engineering and
administered by using policies
Autonomous Network Responsibility Index (ANRI): level of responsibility delegated to the AN in all the Operational
Procedures bind to the lifecycle management of each Autonomous Domain and E2E Service
digital twin: virtual representation of a physical object or system across its lifecycle, using real-time data to
enable understanding, learning and reasoning
®
NOTE: As defined on the IBM website.
domain technical expert: technical expert that has authority within a domain
ETSI

---------------------- Page: 5 ----------------------
6 ETSI GR ENI 010 V1.1.1 (2021-03)
evaluation dimension: factors should be considered in the process of intelligent evaluation
NOTE: As defined in ETSI GR ENI 007 [i.2].
evaluation object: AI application or a part of network lifecycle
Network Digital Twin (NDT): virtual digital twin of telecom network, including its own network lifecycle
NOTE: Some of the dimensions can be tailored or merged in line with actual conditions.
network lifecycle: work-flow of activities including network planning, network deployment, network service
provisioning, network changes, network maintenance, network optimization in real-time
quantitative evaluation criteria: set of rules that can give a score to specific network intelligent application or system
considering multiple dimensions
subsystem: network element, management system, network platform
technical expert: person in charge of defining or supporting Operational Procedures within a CSP Network (e.g. in
charge of Capacity Planning, Engineering and Designing, Troubleshooting)
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the abbreviations given in ETSI GR ENI 004 [i.1] and ETSI
GR ENI 007 [i.2] apply.
4 Introduction
4.1 Background on categories for AI application to networks
At present, artificial intelligence technology has achieved single breakthrough and application in local scene and local
field of network. But there is no unified description language and evolution route of network autonomicity. The
realization of autonomous network needs to evolve step by step in exploration, which cannot be accomplished at a
single stroke. Therefore, a unified standard categories of network autonomicity should be established to measure the
intellectualization level of network and guide the development of network. At present, a variety of network intelligent
grading evaluation systems have been formed in different standards organizations.
Since 2018, ETSI ISG ENI has initiated the network intelligence classification project, officially released in November
2019. On the basis of TMF classification standard, it further describes the characteristics of each level from the
perspectives of market and technology.
The present document will mainly refer to the intelligence grading standard proposed by ETSI ENI and its application
for relevant research and exploration. The definition of categories for AI application to networks is shown in Table 4-1.
Table 4-2 support evaluation of the level of Autonomicity, identifying the responsibility shift from human operator to
the System.
For details, refer to the document ETSI GR ENI 007 [i.2].
ETSI

---------------------- Page: 6 ----------------------
7 ETSI GR ENI 010 V1.1.1 (2021-03)
Table 4-1: Categories of network intelligence from a technical point of view
(Source: ETSI GR ENI 007 [i.2])
Category Name Definition Man-Machine Decision Decision Making Degree of Intelligence Environment Supported
Interface Making and Analysis Adaptability Scenario
Participation
Level 0 Traditional O&M personnel How All-manual Single and shallow Lack of understanding Fixed Single scenario
manual manually control (command) awareness (SNMP (manual understanding
network the network and events and alarms)
obtain network
alarms and logs
Level 1 Partially Automated How Provide Local awareness A small amount of Little change Few scenarios
automated scripts are used (command) suggestions for (SNMP events, analysis
network in service machines or alarms, KPIs, and
automated provisioning, humans and logs)
diagnostics network help decision
deployment, and making
maintenance.
Shallow
perception of
network status
and decision
making
suggestions of
machine
Level 2 Automated Automation of HOW The machine Comprehensive Powerful analysis Little change Few scenarios
network most service (declarative) provides awareness (Telemetry
provisioning, multiple basic data)
network opinions, and
deployment, and the machine
maintenance makes a small
Comprehensive decision
perception of
network status
and local
machine decision
making
Level 3 Self- Deep awareness HOW Most of the Comprehensive and Comprehensive Changeable Multiple
optimization of network status (declarative) machines adaptive sensing (such knowledge scenarios and
network and automatic make as data compression Forecast combinations
network control, decisions and optimization
meeting users' technologies)
network
intentions
ETSI

---------------------- Page: 7 ----------------------
8 ETSI GR ENI 010 V1.1.1 (2021-03)
Category Name Definition Man-Machine Decision Decision Making Degree of Intelligence Environment Supported
Interface Making and Analysis Adaptability Scenario
Participation
Level 4 Partial In a limited WHAT (intent) Optional Adaptive posture Comprehensive Changeable Multiple
autonomous environment, decision- awareness (edge knowledge scenarios and
network people do not making collection + judgment) Forward forecast combinations
need to response
participate in (decision
decision-making comments of
and adjust the challenger)
themselves
Level 5 Autonomous In different WHAT (intent) Machine self- Adaptive deterioration Self-evolution and Any change Any scenario &
network network decision optimization (edge knowledge reasoning combination
environments closed-loop, including
and network collection, judgment,
conditions, the and optimization)
network can
automatically
adapt to and
adjust to meet
people's
intentions

Table 4-2 below referenced from ETSI GR ENI 007 [i.2] reports the level of network autonomicity from a Market point of view, showing the users perception relating to the
business functions of BSS. It is in good alignment with concept defined within TM forum. The scheduling, perception, analysis, customer experience, system capabilities &
network generation may be mapped to technical capabilities. Some like perception and analysis are a one to one mapping. Others, like MMI degree of intelligence and
environment adaptability may each have both a customers and systems aspects.
As reported in clause 5.2 in ETSI GR ENI 007 [i.2] about market relevance: "The factors that impact the market relevance of network autonomicity involve the possibility to
adapt the system and create service offers in different scenarios and involving, according to the 5G network concept, different stakeholders covering a part of or the whole
service chain. The market relevance is determined by aspects as the level of simplicity of the AI assisted Network management, the resulting flexibility of the supported services,
the required effort and staffing to operate and manage the network, the usage of resources and energy, the level of customer experience".
ETSI

---------------------- Page: 8 ----------------------
9 ETSI GR ENI 010 V1.1.1 (2021-03)
Table 4-2: Level of network autonomicity from a market point of view (Source: ETSI GR ENI 007 [i.2])
Level Name Definition Scheduling Perception Analysis and Customer System capability Example of
execution monitoring decision- experience network generation
making
Level 0 Manual O&M O&M operators Operator Operator Operator Operator n/a Command line
manually control the
network and obtain
network alarms and
logs
Level 1 Assisted O&M Automated scripts Operator and Operator Operator Operator Selected service scenarios NMS
are used in service system
provisioning, network
deployment, and
maintenance.
Shallow perception
of network status
and machine
suggestions for
decision making
Level 2 Partial Automation of most Operator and Operator Operator Operator Selected service scenarios NMS + controller
automation service provisioning, System
network deployment,
and maintenance
Comprehensive
perception of
network status and
local machine
decision making
Level 3 Conditional In specific Mostly System Operator and Operator Operator Multiple service scenarios Single-domain:
automation environmental and system Automation + perception
network conditions analysis + limited context-
there is automatic awareness trigger
network control and conditions drive closed-
adaptation loop management
ETSI

---------------------- Page: 9 ----------------------
10 ETSI GR ENI 010 V1.1.1 (2021-03)
Level Name Definition Scheduling Perception Analysis and Customer System capability Example of
execution monitoring decision- experience network generation
making
Level 4 Partial Deep awareness of Mostly System Operator and Operator Operator and Multiple service scenarios Cross-domain (for some
autonomicity network status; in System and System System service scenarios):
most cases the Automation + perception
network performs analysis + experience;
autonomic; context-awareness and
decision-making and simple cognitive
operation adjustment processing closed-loop
management
Level 5 Full In all environmental System System System System Any service scenario Cross-domain and any
autonomicity and network service:
conditions, the Automation + perception
network can analysis + experience;
automatically adapt situation awareness and
cognitive processing
closed-loop management
ETSI

---------------------- Page: 10 ----------------------
11 ETSI GR ENI 010 V1.1.1 (2021-03)
4.2 The motivation for evaluating categories of AI application to
network
Evaluation for categories of AI application to network is proposed to give a score to a specific network intelligent
application considering multiple dimensions (e.g. data collection, analysis, decision, etc.).
Based on the definition of categories and of application cases, according to the use of AI in the implementation process:
1) the technical requirements of each link and step are detailed;
2) the test verification scheme and specification are formulated;
3) the evaluation criteria and index are quantified.
In the evaluation, it is necessary to avoid the requirements for the specific implementation methods of intelligence, and
focus on the evaluation of the implementation effect, such as the degree of automation, whether closed-loop, unit
efficiency, etc.

Figure 4-1: The categorization and evaluation for AI application to network
The definition and evaluation of categories for AI application to network complement each other, jointly promote
network evolution.
The goals and motivation of definition of network autonomicity categories:
• Unified evaluation: Provide basis for categories of network intelligence and promote the whole industry to
form a unified understanding of intelligent network and other related concepts.
• Planning Guideline: Provide reference for operators to formulate relevant strategies, and clarify the stage
division and stage objectives of development planning.
• Decision-making assistance: Provide decision-making assistance for operators, equipment manufacturers and
other industry participants in technology cooperation, product planning, etc.
The goals and motivation of quantitative evaluation criteria:
• Network Evaluation: quantitatively evaluating capability of autonomous network.
• Implementation: defining a process of evaluating network autonomicity categories.
• Technology Innovation: cognizing the disadvantages of the current network and applications, developing new
technologies to improve the level of network autonomicity.
4.3 Responsibility Index in Autonomous Network
Autonomous Network introduces a new aspect to be considered in parallel to the technical capabilities of the Network
and related management systems in themselves. Responsibility and Liability related to autonomous decision represent a
relevant point to be taken into account.
ETSI

---------------------- Page: 11 ----------------------
12 ETSI GR ENI 010 V1.1.1 (2021-03)
The defining characteristic of an Autonomous Network is its ability to assume Responsibilities that the Humans accept
to delegate it.
According to this statement, the level of Autonomicity assumed by network can potentially be regarded less as a
technical one and more related to the decision responsibilities delegated to network by the Operator. Please refer to
Table 4-2 for more information on Operator vs Network responsibilities and roles.
ENI Engine is an enabler for network decision making process across the overall lifecycle of the assisted system.
The Operator, according to AI training and a proper growth in trust for the network capacity to take final decision, can
delegate the responsibility of the decision to AN stepwise.
Any reference in the follow up to the network Responsibility, refers to the level of Responsibility the Operator
delegated to the network to autonomously take the final decision before it get executed.
In some specific case, Human intervention could be needed to execute actions according to decisions taken
automatically by network (e.g. expansion of a datacentre according to a capacity plan generated automatically).
In this case, the Responsibility remains with the network (final decision maker), regardless of the Executor.
In general, responsibility is with the entity taking the ultimate decision, independently of how and who implement the
related actions.
The Operator, in delegating the network for final decision, express trust in network to be properly trained by its experts
and to correctly behave in obtaining expected results. The liability for errors, SLA breach or wrong investment or any
unexpected side effects remain within the Operator remit and is out of scope for the determination of the Autonomicity
Level of the network itself.
Autonomous Network have to control the lifecycle of two main entities: Autonomous Domains and E2E Services.
The Responsibility Level is than strictly related to the level of Autonomicity of the Network in managing the lifecycle
of all its Autonomous Domains and E2E Services.
A quantification of the overall Responsibility Level assumed by the network could be estimated by analysing the
lifecycle and relative Operational phases (network planning, network deployment, network service provisioning,
network changes, network maintenance, network optimization) of each individual Autonomous Domain within the
network, as well as of any E2E Service type.
To properly quantify the Responsibility Level within a Network, Responsibility Matrixes have to be created, having
the phases of the Operator Lifecycle in each column and in each row the Technology Domains (e.g. Transport, Radio,
Fixed Access) or E2E Services (e.g. VoLTE, Enterprise Hybrid Cloud connection, Enterprise VPN).
For each cell of the matrix, a Responsibility Index (e.g. 0 - 5) could be estimated according to:
1) operator responsible of the decision;
2) network has tool to guide and support Operator decision and immediate side effects;
3) network recommend decision presenting a complete view of the element supporting the decision and the
possible side effects;
4) as per level 3, but network has the possibility to take fully autonomous decision in off-peak hours;
5) network fully autonomous in taking decisions, with escalation to Technical Experts in case of severe
unforeseeable events.
The following Tables 4.3 and 4.4 is indicative and modification to lifecycle phases or additions of other Autonomous
Domains or E2E services is possible network planning, network deployment, network service provisioning, network
changes, network maintenance, and network optimization.
ETSI

---------------------- Page: 12 ----------------------
13 ETSI GR ENI 010 V1.1.1 (2021-03)
Table 4-3: Operator Lifecycle Responsibility within Network Domains
Lifecycle/ Auto Total Network planning Network Network service Network changes Network Network
Autonomous DomainWeight deployment provisioning maintenance optimization
Domain
 Score Comment Score Comment Score Comment Score Comment Score Comment Score Comment
Phase Weight X x 1 1 1 1 1 1
(0.1)
RAN 1
Transport 1
Core Network 1
Fixed Access 1
Total

Table 4-4: Operator Lifecycle responsibility within E2E Services
Lifecycle/E2E E2E Total Network planning Network Network service Network changes Network Network optimization
Services Service deployment provisioning maintenance
Weight
Score Comment Score Comment Score Comment Score Comment Score Comment Score Comment
Phase Weight X x 1 1 1 1 1 1
(0.1)
VoLTE 1
Enterprise VPN 1
FWA 1
Enterprise 1
Hybrid Cloud
Connectivity
Total
ETSI

---------------------- Page: 13 ----------------------
14 ETSI GR ENI 010 V1.1.1 (2021-03)
For both the Autonomous Domain and the E2E Services, a partial Autonomous Network Responsibility Index (ANRI)
have to be calculated, respectively ANRItd and ANRIes. For each row a weighted mean value of all the Lifecycle
phases will be calculated. A following weighted mean values of those results will be done on a column bases.
A total ANRI Responsibility Index is t
...