ETSI GR ENI 010 V1.2.1 (2024-06)

Experiential Networked Intelligence (ENI); Evaluation of categories for AI application to Networks

RGR/ENI-0010v121_AI_App_Net

General Information

• Status: Not Published
• Technical Committee: ENI - Experiential Networked Intelligence
• Current Stage: 12 - Completion
• Due Date: 24-Jun-2024
• Completion Date: 26-Jun-2024

Standards Content (Sample)

GROUP REPORT
Experiential Networked Intelligence (ENI);
Evaluation of categories for AI application to Networks
Disclaimer
The present document has been produced and approved by the Experiential Networked Intelligence (ENI) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

2 ETSI GR ENI 010 V1.2.1 (2024-06)

Reference
RGR/ENI-0010v121_AI_App_Net
Keywords
artificial intelligence, categorization, network
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - APE 7112B
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° w061004871

Important notice
The present document can be downloaded from:
https://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
If you find a security vulnerability in the present document, please report it through our
Coordinated Vulnerability Disclosure Program:
https://www.etsi.org/standards/coordinated-vulnerability-disclosure
Notice of disclaimer & limitation of liability
The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of
experience to understand and interpret its content in accordance with generally accepted engineering or
other professional standard and applicable regulations.
No recommendation as to products and services or vendors is made or should be implied.
No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law
rule and/or regulation and further, no representation or warranty is made of merchantability or fitness
and/or governmental
for any particular purpose or against infringement of intellectual property rights.
In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.

Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not
limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property
rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages
for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use
of or inability to use the software.
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and
microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2024.
All rights reserved.
ETSI
3 ETSI GR ENI 010 V1.2.1 (2024-06)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
1 Scope . 5
2 References . 5
2.1 Normative references . 5
2.2 Informative references . 5
3 Definition of terms, symbols and abbreviations . 5
3.1 Terms . 5
3.2 Symbols . 6
3.3 Abbreviations . 6
4 Introduction . 6
4.1 Background on categories for AI application to networks . 6
4.2 The motivation for evaluating categories of AI application to network . 11
4.3 Responsibility Index in Autonomous Network . 11
5 Evaluation criteria of categories for AI application to Network . 14
5.1 Framework of quantitative evaluation process . 14
5.2 Scoring principles and specification of the single scenario . 16
5.2.1 Scoring principles and specification . 16
5.2.2 Weights determined by Analytic Hierarchy Process . 17
5.2.3 Recommended values of KPI for each intelligent level in some scenarios . 19
5.2.4 Presentation of evaluation result . 21
5.3 Scoring principles and specification of the part of network lifecycle . 21
5.3.1 Explorations on evaluation of network deployment autonomicity categories . 21
5.3.2 Explorations on evaluation of evaluation of the entire network autonomicity categories . 24
5.4 The relationship among the network infrastructure capabilities (KPI) and the network intelligence levels . 29
6 Application of quantitative evaluation criteria . 30
6.1 Transport Network: 5G and DCN examples . 30
6.1.1 General categorization criteria . 30
6.1.2 5G transport example . 34
6.1.3 DCN transport example . 36
6.1.4 IP Network Monitoring and Troubleshooting Process . 37
7 Suggested requirements of network infrastructure in different Categories . 43
7.1 Knowledge Base . 43
7.2 Tool for decision delegation according to Time Dimension . 43
7.3 Data classification and grading . 44
8 Conclusions . 45
Annex A: Change history . 46
History . 47

ETSI
4 ETSI GR ENI 010 V1.2.1 (2024-06)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations
pertaining to these essential IPRs, if any, are publicly available for ETSI members and non-members, and can be
found in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to
ETSI in respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the
ETSI Web server (https://ipr.etsi.org/).
Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs,
including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not
referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become,
essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its

Members. 3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and of the 3GPP
Organizational Partners. oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and of the ®
oneM2M Partners. GSM and the GSM logo are trademarks registered and owned by the GSM Association.
Foreword
This Group Report (GR) has been produced by ETSI Industry Specification Group (ISG) Experiential Networked
Intelligence (ENI).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI
5 ETSI GR ENI 010 V1.2.1 (2024-06)
1 Scope
The present document revises ETSI GR ENI 010 [i.4] to further:
• investigate quantitative evaluation criteria of network autonomicity categories;
• perform a deeper research of more quantitative factors that determine those categories;
• define an accurate scoring criteria that complies with the evolution of the ENI architecture; and
• define a data model covering an entire operator's network or just a specific domain.
This deeper research will be complemented by the description of several example scenarios where the quantitative
factors and the scoring evaluation criteria will be illustrated. This can be done by analysing the relationship among
network KPIs of different levels, e.g. between the network infrastructure capabilities and the network intelligence
levels.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] Void.
[i.2] ETSI GR ENI 004 (V3.1.1): "Experiential Networked Intelligence (ENI); Terminology".
[i.3] ETSI GR ENI 007 (V1.1.1): "Experiential Networked Intelligence (ENI); ENI Definition of
Categories for AI Application to Networks".
[i.4] ETSI GR ENI 010 (V1.1.1): "Experiential Networked Intelligence (ENI); Evaluation of categories
for AI application to Networks".
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
autonomous networks: set of self-governing programmable and explainable systems that seamlessly deliver secure,
context-aware, business-driven services that are created and maintained using model-driven engineering and
administered by using policies
ETSI
6 ETSI GR ENI 010 V1.2.1 (2024-06)
Autonomous Network Responsibility Index (ANRI): level of responsibility delegated to the AN in all the Operational
Procedures bind to the lifecycle management of each Autonomous Domain and E2E Service
digital twin: virtual representation of a physical object or system across its lifecycle, using real-time data to
enable understanding, learning and reasoning ®
NOTE: As defined on the IBM website.
domain technical expert: technical expert that has authority within a domain
evaluation dimension: viewpoint that can be divided into five dimensions
NOTE: This can be subdivided into Decision Making Participation, Data Collection and Analysis, Degree of
Intelligence and Environment Adaptability, as defined in ETSI GR ENI 007 [i.3].
evaluation object: AI application or a part of Network Lifecycle, defined from two dimensions: the subsystems and the
network lifecycle
Network Digital Twin (NDT): Virtual Digital Twin of telecom network, including its own Network lifecycle
NOTE: Some of the dimensions can be tailored or merged in line with actual conditions.
network lifecycle: work-flow of activities including network planning, network deployment, network service
provisioning, network changes, network maintenance, network optimization in real-time
quantitative evaluation criteria: give a score to specific network intelligent application or system considering multiple
dimensions
subsystem: network element, management system, network platform
technical expert: person in charge of defining or supporting Operational Procedures within a CSP Network (e.g. in
charge of Capacity Planning, Engineering and Designing, Troubleshooting)
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the abbreviations given in ETSI GR ENI 004 [i.2] and ETSI
GR ENI 007 [i.3] apply.
4 Introduction
4.1 Background on categories for AI application to networks
At present, artificial intelligence technology has achieved single breakthrough and application in local scene and local
field of network. But there is no unified description language and evolution route of network autonomicity. The
realization of autonomous network needs to evolve step by step in exploration, which cannot be accomplished at a
single stroke. Therefore, a unified standard categories of network autonomicity should be established to measure the
intellectualization level of network and guide the development of network. At present, a variety of network intelligent
grading evaluation systems have been formed in different standards organizations.
Since 2018, ETSI ISG ENI has initiated the network intelligence classification project, officially released in
November 2019. On the basis of TMF classification standard, it further describes the characteristics of each level from
the perspectives of market and technology.
The present document will mainly refer to the intelligence grading standard proposed by ETSI ENI and its application
for relevant research and exploration. The definition of categories for AI application to networks is shown in Table 4-1.
ETSI
7 ETSI GR ENI 010 V1.2.1 (2024-06)
Table 4-2 support evaluation of the level of Autonomicity, identifying the responsibility shift from human operator to the System.
For details, refer to ETSI GR ENI 007 [i.3].
Table 4-1: Categories of network intelligence from a technical point of view
(Source: ETSI GR ENI 007 [i.3])
Category Name Definition Man-Machine Decision Decision Making Degree of Intelligence Environment Supported
Interface Making and Analysis Adaptability Scenario
Participation
Level 0 Traditional O&M personnel How All-manual Single and shallow Lack of understanding Fixed Single scenario
manual manually control (command) awareness (SNMP (manual understanding
network the network and events and alarms)
obtain network
alarms and logs
Level 1 Partially Automated How Provide Local awareness A small amount of Little change Few scenarios
automated scripts are used (command) suggestions for (SNMP events, analysis
network in service machines or alarms, KPIs and logs)
automated provisioning, humans and
diagnostics network help decision
deployment, and making
maintenance.
Shallow
perception of
network status
and decision
making
suggestions of
machine
Level 2 Automated Automation of HOW The machine Comprehensive Powerful analysis Little change Few scenarios
network most service (declarative) provides awareness (Telemetry
provisioning, multiple basic data)
network opinions, and
deployment, and the machine
maintenance makes a small
Comprehensive decision
perception of
network status
and local
machine decision
making
ETSI
8 ETSI GR ENI 010 V1.2.1 (2024-06)
Category Name Definition Man-Machine Decision Decision Making Degree of Intelligence Environment Supported
Interface Making and Analysis Adaptability Scenario
Participation
Level 3 Self- Deep awareness HOW Most of the Comprehensive and Comprehensive Changeable Multiple
optimization of network status (declarative) machines adaptive sensing (such knowledge scenarios and
network and automatic make as data compression Forecast combinations
network control, decisions and optimization
meeting users' technologies)
network
intentions
Level 4 Partial In a limited WHAT (intent) Optional Adaptive posture Comprehensive Changeable Multiple
autonomous environment, decision- awareness (edge knowledge scenarios and
network people do not making collection + judgment) Forward forecast combinations
need to response
participate in (decision
decision-making comments of
and adjust the challenger)
themselves
Level 5 Autonomous In different WHAT (intent) Machine self- Adaptive deterioration Self-evolution and Any change Any scenario &
network network decision optimization (edge knowledge reasoning combination
environments closed-loop, including
and network collection, judgment,
conditions, the and optimization)
network can
automatically
adapt to and
adjust to meet
people's
intentions
ETSI
9 ETSI GR ENI 010 V1.2.1 (2024-06)
Table 4-2 below referenced from ETSI GR ENI 007 [i.3] report the level of network autonomicity from a Market point of view, showing the users perception relating to the
business functions of BSS. It is in good alignment with concept defined within TMforum. The scheduling, perception, analysis, customer experience, system capabilities &
network generation may be mapped to technical capabilities. Some like perception and analysis are a one to one mapping. Others, like MMI degree of intelligence and
environment adaptability may each have both a customers and systems aspects.
As reported in clause 5.2 in ETSI GR ENI 007 [i.3] about market relevance: "The factors that impact the market relevance of network autonomicity involve the possibility to
adapt the system and create service offers in different scenarios and involving, according to the 5G network concept, different stakeholders covering a part of or the whole
service chain. The market relevance is determined by aspects as the level of simplicity of the AI assisted Network management, the resulting flexibility of the supported
services, the required effort and staffing to operate and manage the network, the usage of resources and energy, the level of customer experience".
The 6 levels as described below are an ENI view.
Table 4-2: Level of network autonomicity from a market point of view (Source: ETSI GR ENI 007 [i.3])
Level Name Definition Scheduling Perception Analysis and Customer System capability Example of
execution monitoring decision- experience network generation
making
Level 0 Manual O&M O&M operators Operator Operator Operator Operator n/a Command line
manually control the
network and obtain
network alarms and
logs
Level 1 Assisted O&M Automated scripts Operator and Operator Operator Operator Selected service scenarios NMS
are used in service system
provisioning, network
deployment, and
maintenance.
Shallow perception
of network status
and machine
suggestions for
decision making
Level 2 Partial Automation of most Operator and Operator Operator Operator Selected service scenarios NMS + controller
automation service provisioning, System
network deployment,
and maintenance
Comprehensive
perception of
network status and
local machine
decision making
ETSI
10 ETSI GR ENI 010 V1.2.1 (2024-06)
Level Name Definition Scheduling Perception Analysis and Customer System capability Example of
execution monitoring decision- experience network generation
making
Level 3 Conditional In specific Mostly System Operator and Operator Operator Multiple service scenarios Single-domain:
automation environmental and system Automation + perception
network conditions analysis + limited context-
there is automatic awareness trigger
network control and conditions drive closed-
adaptation loop management
Level 4 Partial Deep awareness of Mostly System Operator and Operator Operator and Multiple service scenarios Cross-domain (for some
autonomicity network status; in System and System System service scenarios):
most cases the Automation + perception
network performs analysis + experience;
autonomic; context-awareness and
decision-making and simple cognitive
operation adjustment processing closed-loop
management
Level 5 Full In all environmental System System System System Any service scenario Cross-domain and any
autonomicity and network service:
conditions, the Automation + perception
network can analysis + experience;
automatically adapt situation awareness and
cognitive processing
closed-loop management
ETSI
11 ETSI GR ENI 010 V1.2.1 (2024-06)
4.2 The motivation for evaluating categories of AI application to
network
Evaluation for categories of AI application to network is proposed to give a score to a specific network intelligent
application considering multiple dimensions (e.g. data collection, analysis, decision, etc.).
Based on the definition of categories and of application cases, according to the use of AI in the implementation process:
1) the technical requirements of each link and step are detailed;
2) the test verification scheme and specification are formulated;
3) the evaluation criteria and index are quantified.
In the evaluation, it is necessary to avoid the requirements for the specific implementation methods of intelligence, and
focus on the evaluation of the implementation effect, such as the degree of automation, whether closed-loop, unit
efficiency, etc.
Figure 4-1: The categorization and evaluation for AI application to network
The definition and evaluation of categories for AI application to network complement each other, jointly promote
network evolution.
The goals and motivation of definition of network autonomicity categories:
• Unified evaluation: Provide basis for categories of network intelligence and promote the whole industry to
form a unified understanding of intelligent network and other related concepts.
• Planning Guideline: Provide reference for operators to formulate relevant strategies, and clarify the stage
division and stage objectives of development planning.
• Decision-making assistance: Provide decision-making assistance for operators, equipment manufacturers and
other industry participants in technology cooperation, product planning, etc.
The goals and motivation of quantitative evaluation criteria:
• Network Evaluation: quantitatively evaluating capability of autonomous network.
• Implementation: defining a process of evaluating network autonomicity categories.
• Technology Innovation: cognizing the disadvantages of the current network and applications, developing new
technologies to improve the level of network autonomicity.
4.3 Responsibility Index in Autonomous Network
Autonomous Network introduce a new aspect to be considered in parallel to the technical capabilities of the Network
and related management systems in themselves. Responsibility and Liability related to autonomous decision represent a
relevant point to be taken into account.
ETSI
12 ETSI GR ENI 010 V1.2.1 (2024-06)
The defining characteristic of an Autonomous Network is its ability to assume Responsibilities that the Humans accept
to delegate it.
According to this statement, the level of Autonomicity assumed by network can potentially be regarded less as a
technical one and more related to the decision responsibilities delegated to network by the Operator. Refer to Table 4-2
for more information on Operator vs Network responsibilities and roles.
ENI Engine is an enabler for network decision making process across the overall lifecycle of the assisted system.
The Operator, according to AI training and a proper growth in trust for the network capacity to take final decision, can
delegate the responsibility of the decision to AN stepwise.
Any reference in the follow up to the network Responsibility, refers to the level of Responsibility the Operator
delegated to the network to autonomously take the final decision before it get executed.
In some specific case, Human intervention could be needed to execute actions according to decisions taken
automatically by network (e.g. expansion of a datacentre according to a capacity plan generated automatically).
In this case, the Responsibility remains with the network (final decision), regardless of the Executor.
In general, responsibility is with the entity taking the ultimate decision, independently of how and who implement the
related actions.
The Operator, in delegating the network for final decision, express trust in network to be properly trained by its experts
and to correctly behave in obtaining expected results. The liability for errors, SLA breach or wrong investment or any
unexpected side effects remain within the Operator remit and is out of scope for the determination of the Autonomicity
Level of the network itself.
Autonomous Network have to control the lifecycle of two main entities: Autonomous Domains and E2E Services.
The Responsibility Level is than strictly related to the level of Autonomicity of the Network in managing the lifecycle
of all its Autonomous Domains and E2E Services.
A quantification of the overall Responsibility Level assumed by the network could be estimated by analysing the
lifecycle and relative Operational phases (network planning, network deployment, network service provisioning,
network changes, network maintenance, network optimization) of each individual Autonomous Domain within the
network, as well as of any E2E Service type.
To properly quantify the Responsibility Level within a Network, Responsibility Matrixes have to be created, having the
phases of the Operator Lifecycle in each column and in each row the Technology Domains (e.g. Transport, Radio,
Fixed Access) or E2E Services (e.g. VoLTE, Enterprise Hybrid Cloud connection, Enterprise VPN).
For each cell of the matrix, a Responsibility Index (e.g. 0-5) could be estimated according to:
1) operator responsible of the decision;
2) network has tool to guide and support Operator decision and immediate side effects;
3) network recommend decision presenting a complete view of the element supporting the decision and the
possible side effects;
4) as per level 3, but network has the possibility to take fully autonomous decision in off-peak hours;
5) network fully autonomous in taking decisions, with escalation to Technical Experts in case of severe
unforeseeable events.
The following Tables 4-3 and 4-4 are indicative and modification to lifecycle phases or additions of other Autonomous
Domains or E2E services is possible network planning, network deployment, network service provisioning, network
changes, network maintenance, and network optimization.
ETSI
13 ETSI GR ENI 010 V1.2.1 (2024-06)
Table 4-3: Operator Lifecycle Responsibility within Network Domains
Lifecycle/ Auto Total Network planning Network Network service Network changes Network Network
Autonomous DomainWeight deployment provisioning maintenance optimization
Domain
Score Comment Score Comment Score Comment Score Comment Score Comment Score Comment
Phase Weight X x 1 1 1 1 1 1
(0.1)
RAN 1
Transport 1
Core Network 1
Fixed Access 1
Total
Table 4-4: Operator Lifecycle responsibility within E2E Services
Lifecycle/E2E E2E Total Network planning Network Network service Network changes Network Network optimization
Services Service deployment provisioning maintenance
Weight
Score Comment Score Comment Score Comment Score Comment Score Comment Score Comment
Phase Weight X x 1 1 1 1 1 1
(0.1)
VoLTE 1
Enterprise VPN 1
FWA 1
Enterprise 1
Hybrid Cloud
Connectivity
Total
ETSI
14 ETSI GR ENI 010 V1.2.1 (2024-06)
For both the Autonomous Domain and the E2E Services, a partial Autonomous Network Responsibility Index (ANRI)
have to be calculated, respectively ANRItd and ANRIes. For each row a weighted mean value of all the Lifecycle
phases will be calculated. A following weighted mean values of those results will be done on a column bases.
A total ANRI Responsibility Index is than calculated with a weighted mean value of the two value above:
��� � ������ � ��� � ������
�����
�������
The calculated (ANRI) could be an additional component of the Score S determined in Table 5-2.
On the other side, due to the completeness of the evaluation required to evaluate the ANRI, it can be considered as the
achieved level of Autonomicity achieved by the network itself.
ANRI is a methodology for calculating the � (score of Decision Making Participation) as defined and used in
���
clause 5.2.
5 Evaluation criteria of categories for AI application to
Network
5.1 Framework of quantitative evaluation process
The general process of evaluating categories of AI application to network includes five steps: the identification of the
evaluation object, the division of the evaluation dimension, the analysis of the evaluation object, the scoring of the
evaluation dimension and the acquisition of the evaluation result, as shown in Figure 5-1.

Figure 5-1: Framework of quantitative evaluation process
The specific process is as follows:
1) The identification of the evaluation object
When the evaluation object is selected from the actual production system, it needs to be defined from two
dimensions: the end to end subsystems and network lifecycle, so as to better analyse the corresponding
quantitative indicators. Some examples are given in Table 5-1.
ETSI
15 ETSI GR ENI 010 V1.2.1 (2024-06)
Table 5-1: Examples of evaluation object
Network Network
Network Network Service Network Network
Planning and Changes
Deployment Providing Maintenance Optimization
Design
Intelligent Network  Network MM Parameter
Network Hardware Element Element Fault Optimization and
Element
Recognition Cutover Location Base Station
Energy Saving
Site Planning Network Service Node upgrade Wireless Network
Management
Tool Expansion Tool Management management Optimization Tool
System
System
The Platform The Platform of Operation Network Intelligent Intelligent
Network
of Network Device Online Support System upgrade Monitoring Network
Platform Planning management System Optimization
System System
2) The Division of the Evaluation Dimension
According to Table 4-1, when evaluating an object, it can be divided into five dimensions such as
ManMachine Interface, Decision Making Participation, Data Collection and Analysis, Degree of intelligence
and Environment adaptability or some of the dimensions can be tailored or merged in line with actual
conditions.
3) The Analysis of the Evaluation Object
After defining the evaluation dimensions of the evaluation object, each evaluation dimension can be divided
into the following indices. Information extraction and status analysis are carried out for each index, so as to
realize the quantification of each index and to support the scoring in step 4.
Table 5-2: Analysis of evaluation objects
ManMachine Decision Making Data Collection and Degree of Environment
Interface (MMI) Participation (DMP) Analysis (DCA) Intelligence (DI) Adaptability (EA)
User Decision-making - Collection Content Analysis Content Robustness index
Requirements Content
Interface mode Decision-making Collection Methods Analysis Methods Adaptation mode
Methods
System Decision-making Collection Results Analysis Results Adaptation
Requirements Results Result/Time

NOTE 1: The accuracy of adaptive time division needs further study.
NOTE 2: How to quantify environmental change needs further study.
4) The Scoring of the Evaluation Dimension
In accordance with the scoring principles in clause 5.2.1, after obtaining the detailed status of each dimension
in step 3, each dimension will be scored.
5) The Acquisition of the Evaluation Result
At the end of the scoring of each dimension, it is necessary to perform a weighted calculation based on each
score to obtain the score of the entire evaluated object. The presentation of the evaluation results is described
in clause 5.2.3.
ETSI
16 ETSI GR ENI 010 V1.2.1 (2024-06)
5.2 Scoring principles and specification of the single scenario
5.2.1 Scoring principles and specification
According to the evaluating process defined, the evaluation of categories for single scenario involves two parts:
1) Scoring of each evaluation dimension (� , � , S , � , � ):
��� ��� ��� �� ��
The purpose of this step is to complete the scoring of ManMachine Interface, Decision Making Participation,
Data Collection and Analysis, Degree of Intelligence and Environment Adaptability, and lay the foundation for
the overall scoring of the evaluation object. Note that inevitable subjective factors will influence the
evaluation.
According to Table 4-1, the score of each dimension can be divided into six categories: S=0, S=1, S=2, S=3,
S=4, S=5. The scoring rules are as follows:
- � : means the score of ManMachine Interface. According to Tables 5-1 and 5-2, Interface Mode is the
���
index that has the greatest impact on the score of this dimension. For example, when the interface mode
is command, the dimension score is S=0 or S=1. But the more accurate score can be determined by two
other indices: User Requirements and System Requirements. Obviously, when S=1, the requirements
generated by the system should be greater than the user's, or the system can better understand the
command.
- � : means the score of Decision Making Participation. Decision-making Methods is the index that has
���
the greatest impact on the score of this dimension. When the proportion of work done by the system is
0 %, S=0; When the proportion of work done by the system is less than 50 %, S=1; When the proportion
of work done by the system is 50 %~75 %, S1=2; When the proportion of work done by the system is
75 %~90 %, S=3; When the proportion of work done by the system is more than 90 %, S=4. When the
proportion of work done by the system is 100 %, S=5. The percentage can be determined by two other
indices of the evaluated object: Decision-making Dimension and Decision-making Results.
- S : means the score of Data Collection and Analysis. Collection Methods is the index that has the
���
greatest impact on the score of this dimension. When the proportion of work done by the system is 0 %,
S=0; When the proportion of work done by the system is less than 50 %, S=1; When the proportion of
work done by the system is 50 %~75 %, S1=2; When the proportion of work done by the system is
75 %~90 %, S=3; When the proportion of work done by the system is more than 90 %, S=4. When the
proportion of work done by the system is 100 %, S=5. The percentage can be determined by two other
indices of the evaluated object: Collection Content and Collection Results.
- � : means the score of Degree of Intelligence. Analysis Methods is the index that has the greatest impact
��
on the score of this dimension. When the proportion of work done by the system is 0 %, S=0; When the
proportion of work done by the system is less than 50 %, S=1; When the proportion of work done by the
system is 50 %~75 %, S1=2; When the proportion of work done by the system is 75 %~90 %, S=3;
When the proportion of work done by the system is more than 90 %, S=4. When the proportion of work
done by the system is 100 %, S=5. The percentage can be determined by two other indices of the
evaluated object: Analysis Content and Analysis Results.
- � : means the score of Environment Adaptability.
��
NOTE: How to quantify environmental change needs further study.
2) The overall scoring of evaluation object (�):
After completing the evaluation of each dimension, the overall score of the evaluation object can be completed
based on the following formula:
S= w × � +w ×S +� ×� +� ×� +� ×�
��� ��� ��� ��� ��� ��� �� �� �� ��
w means the weight of ManMachine Interface.
���
w means the weight of Decision Making Participation in overall scoring.
���
w means the weight of Data Collection and Analysis in overall scoring.
���
ETSI
17 ETSI GR ENI 010 V1.2.1 (2024-06)
w means the weight of Degree of Intelligence in overall scoring.
��
w means the weight of Environment Adaptability in overall scoring.
��
The weight can be determined by the following methods:
- Expert experience.
- From the perspective of evaluation dimension, it can be determined by some fuzzy quantization
algorithms, e.g. the Analytic Hierarchy Process (AHP) introduced in clause 5.2.2.
- From the perspective of key effect indicators of evaluation objects, the weight is determined by analysing
the influence of dimensions on effect indicators, e.g. accuracy, real-time, unit income, etc.
The corresponding relation between network autonomicity categories and overall score is shown in Table 5-3.
Table 5-3: The corresponding relation between categories and overall Score
Category Score (�)
L0 ���
L1 �����
L2 �����
L3 �����
L4 �����
L5 ���
5.2.2 Weights determined by Analytic Hierarchy Process
Analytic Hierarchy Process (AHP) is a decision analysis method that combines qualitative and quantitative methods to
solve complex multi-objective problems. The specific steps of applying this method to evaluate network autonomicity
categories are as follows:
1) Establish a hierarchical model
Firstly, according to the relationship among the decision alternative (categories), decision criteria (dimension)
and target (evaluation objects), it is divided into the highest level, the middle level and the lowest level, as
shown in Figure 5-2:
- The highest level refers to the purpose of the decision or the problem to be solved. Here, the verification
goal is the autonomicity categories of the evaluation object.
- The middle level refers to the factors to be considered or the criteria for decision making. Here, it refers
to the evaluation dimensions.
- The lowest level refers to the alternatives for decision making, here it refers to categories L0 ~ L5.

Figure 5-2: Hierarchy chart of network autonomicity categories
ETSI
18 ETSI GR ENI 010 V1.2.1 (2024-06)
2) Constructing a judgment(paired comparison) matrix A
- Expert experience is used to judge the importance of each criterion relative to the target, the
quantification value of the importance degree of each criterion(dimensions) compared with other criteria
is given reasonably, which can be obtained through Table 5-4.
Table 5-4: Quantitative value of the importance of each dimension
Comparison between criterion I and criterion J Quantization value
Equally important 1
Slightly important 3
More important 5
Strongly important 7
Extremely important 9
Intermediate value of two adjacent judgments 2, 4, 6, 8

- The weights are arranged in order to construct the following judgment (paired comparison) matrix:

aaa a a
mmi__mmi mmi dmp mmi_ dca mmi_ di mmi_ ea

aaa a a

dmp___mmi dmp dmp dmp dca dmp_ di dmp_ ea

Aa= a a a a

dca__mmi dca dmp dca_ dca dca_ di dca_ ea

aa a a a
di__mmi di dmp di_ dca di_ di di_ ea


aa a a a
ea__mmi eadmp ea_dca ead_ i eae_ a

a means the quantization value of comparison between ManMachine Interface and
���_���
ManMachine Interface.
a means the quantization value of comparison between ManMachine Interface and
���_���
Decision Making Participation.
a means the quantization value of comparison between ManMachine Interface and Data
���_���
Collection and Analysis.
a means the quantization value of comparison between ManMachine Interface and Degree of
���_��
Intelligence.
a means the quantization value of comparison between ManMachine Interface and
���_��
Environment Adaptability. Others and so on.
3) Consistency test and weight determination
The eigenvector corresponding to the largest eigenvalue (� ) of the judgment matrix is normalized to W.
���
��
Generally, when consistency ratio CR = <0,1, it is considered that the degree of inconsistency of A is
��
within the allowable range, with satisfactory consistency, and passes the consistency test. The final weight
vector can be obtained by its normalized eigenvector:
W =ww,,w,w,w
{ }
ac da an
...