ETSI GR CIM 030 V1.1.1 (2023-12)

Context Information Management (CIM); Validation of NGSI-LD test Platform and Examples of uses

DGR/CIM-0030

General Information

• Status: Not Published
• Technical Committee: CIM - cross-cutting Context Information Management
• Current Stage: 12 - Citation in the OJ (auto-insert)
• Due Date: 27-Nov-2023
• Completion Date: 01-Dec-2023

Standards Content (Sample)

GROUP REPORT
Context Information Management (CIM);
Validation of NGSI-LD test Platform and Examples of uses
Disclaimer
The present document has been produced and approved by the cross-cutting Context Information Management (CIM) ETSI
Industry Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.

2 ETSI GR CIM 030 V1.1.1 (2023-12)

Reference
DGR/CIM-0030
Keywords
data interoperability, data models, data sharing,
NGSI-LD, PCTR, testing
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - APE 7112B
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° w061004871

Important notice
The present document can be downloaded from:
https://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
If you find a security vulnerability in the present document, please report it through our
Coordinated Vulnerability Disclosure Program:
https://www.etsi.org/standards/coordinated-vulnerability-disclosure
Notice of disclaimer & limitation of liability
The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of
experience to understand and interpret its content in accordance with generally accepted engineering or
other professional standard and applicable regulations.
No recommendation as to products and services or vendors is made or should be implied.
No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law
rule and/or regulation and further, no representation or warranty is made of merchantability or fitness
and/or governmental
for any particular purpose or against infringement of intellectual property rights.
In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.

Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not
limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property
rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages
for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use
of or inability to use the software.
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and
microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2023.
All rights reserved.
ETSI
3 ETSI GR CIM 030 V1.1.1 (2023-12)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definition of terms, symbols and abbreviations . 6
3.1 Terms . 6
3.2 Symbols . 6
3.3 Abbreviations . 7
4 NGSI-LD Interface Test Validation . 7
4.1 Introduction . 7
4.2 Test Environment . 7
4.2.1 Implementation Under Test (IUT) . 7
4.2.2 Test Equipment . 8
4.2.3 Test Environment . 8
4.2.4 Test case . 9
4.3 Update history about NGSI-LD interface test code . 10
4.3.1 Introduction. 10
4.3.2 Test cases Design . 10
4.3.3 Validation . 11
4.3.4 Validation Result . 11
4.3.5 Update History . 14
4.3.5.0 Introduction . 14
4.3.5.1 Deletion of Unnecessary Retrieve Request for Test Purpose . 14
4.3.5.2 datasetid . 14
4.3.5.3 Mismatch between Test Standards and Test equipment . 14
5 NGSI-LD based Data Model Test Validation . 15
5.1 Introduction . 15
5.2 Test Environment . 15
5.2.1 Implementation Under Test . 15
5.2.2 Test Equipment . 15
5.2.3 Test Environment . 17
5.3 Test case . 17
5.3.1 Valid Behaviour (BV) . 18
5.3.2 Invalid Behaviour (BI) . 19
6 Test and Validation case studies . 21
6.1 Introduction . 21
6.2 Validation result of Data Hub in Daegu city . 21
6.2.1 Interface . 21
6.2.2 Data Model . 23
6.3 Validation result of Data Hub in Siheung city. 24
6.3.1 Interface . 24
6.3.2 Data Model . 27
7 Future Studies . 28
7.1 Introduction . 28
7.2 Requirement for Interoperability in Smart City Data . 29
7.2.1 Data Unity for Data Sharing and Interoperability . 29
7.2.2 Control Tower for Interoperability of Data . 29
7.2.3 Integrated Management System from Creation to Disposal of Data . 29
7.3 Guidelines for Securing Interoperability of Smart City Data . 29
ETSI
4 ETSI GR CIM 030 V1.1.1 (2023-12)
7.3.1 Establishment of Data Management System for Integrated Data Management . 29
7.3.2 Establishing a Data Identification System through Global Uniqueness of City Data and Objects . 30
7.3.3 Developing Governance for City Data System and Policy Establishment . 31
7.4 Utilization of NGSI-LD for data interoperability . 31
Annex A: Change history . 32
History . 33

ETSI
5 ETSI GR CIM 030 V1.1.1 (2023-12)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations
pertaining to these essential IPRs, if any, are publicly available for ETSI members and non-members, and can be
found in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to
ETSI in respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the
ETSI Web server (https://ipr.etsi.org/).
Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs,
including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not
referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become,
essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its

Members. 3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and of the 3GPP
Organizational Partners. oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and of the ®
oneM2M Partners. GSM and the GSM logo are trademarks registered and owned by the GSM Association.
Foreword
This Group Report (GR) has been produced by ETSI Industry Specification Group (ISG) cross-cutting Context
Information Management (CIM).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI
6 ETSI GR CIM 030 V1.1.1 (2023-12)
1 Scope
The present document provides the case study about validation of NGSI-LD test platform and describe data
interoperability in point of NGSI-LD interface. The case study describes how to develop the NGSI-LD interface and
data model test platform. At the conclusion, the present document describe how the NGSI-LD test platform can be used
for data interoperability in Smart city.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GS CIM 009 (V1.6.1): "cross-cutting Context Information Management (CIM); NGSI-LD
API".
[i.2] ETSI GS CIM 012 (V1.1.1): "Context Information Management (CIM); NGSI-LD Test Suite
Structure".
[i.3] ETSI GS CIM 013: "Context Information Management (CIM); NGSI-LD Test Purposes
Descriptions".
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
Data Model: model representing the structure of data stored and managed in a data hub, indicating the abstract model
required during the access and processing of data
Data Model schema: specification that defines the data model utilized in a Data Hub, outlining the overall
specifications for structuring and constraining data in order to objectify and load it into a database for utilization
3.2 Symbols
Void.
ETSI
7 ETSI GR CIM 030 V1.1.1 (2023-12)
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Process Interface
ATS Abstract Test Suite
BI Invalid Behaviour tests
BV Behaviour Valid tests
CCTV Closed-Circuit Television
CPS Cyber Physical System
GS1 Global Standard one
ICS Implementation Conformance Statement
ID Identity or Identifier
IUT Implementation Under Test
IXIT Implementation eXtra Information for Test
JSON JavaScript Object Notation
KETI Korea Electronics Technology Institution
NGSI-LD Next Generation Service Interfaces Linked Data
QR Quick Response code
RFID Radio Frequency Identification
SUT System - Under Test
TC Test Case
TP Test Purpose
TTA Telecommunications Technology Association (South Korea)
URI Unified Resource Identifier
URL Uniform Ressource Locator
4 NGSI-LD Interface Test Validation
4.1 Introduction
Korea Electronics Technology Institute (KETI) developed a Data Hub by adopting ETSI GS CIM 009 [i.1]. Moreover,
ETSI CIM developed a test framework and test suite that can test the NGSI-LD standard conformance and released a
develop version of the test suite. The test suite is a development version and in order to apply it to the test equipment, it
is necessary to secure the validity of the test operation, additional implementation, and validation of the implementation.
This session introduces the process of establishing the test system such as the test scope, test equipment, and test
environment, focusing on the interface functions supported by the Data Hub.
4.2 Test Environment
4.2.1 Implementation Under Test (IUT)
The smart city Data Hub divides logical functions into its modules and connects with each internal module or external
service through the NGSI-LD interface to send request and responses to operations. Among the modules of the Data
Hub, its core module stores smart city data based on the NGSI-LD information model and exchanges data between
internal modules and external systems through the NGSI-LD interface. In this session, the data core module responsible
for core functions in the Data Hub was defined as an IUT, and the interface conformance test was performed by
adopting ETSI GS CIM 009 [i.1] and test framework. The Data core module includes a Context Broker supporting
NGSI-LD interface and the Context Broker has a data structure based on the NGSI-LD information model. The Context
Broker performs data modelling through context-based definitions of common terms.
ETSI
8 ETSI GR CIM 030 V1.1.1 (2023-12)

Figure 4.2.1-1: IUT system architecture for NGSI-LD interface test
4.2.2 Test Equipment
In order to set up Data Hub NGSI-LD interface conformance test, test equipment was developed based on technical
specifications if required. The test equipment developed by complying with the standards defined by ETSI
GS CIM 009 [i.1] and test framework has test procedures and criteria for each test case. The test equipment requests
Create, retrieve, update, delete, and subscription operations for the data stored by the IUT, and the IUT returns a
response message to the test equipment in response to the requested operation.

Figure 4.2.2-1: NGSI-LD Interface Test equipment
For the development of the test equipment, test cases were derived according to the interface functions supported by the
Data Hub and the test suite was applied at the source code level. Figure 4.2.2-1 shows the system architecture and user
interface of test equipment. The test equipment performs test planning, test execution, and test result reporting functions
through the test manager and presents a PASS or Fail decision for test results for each test case in connection with the
test suite of ETSI CIM. ETSI CIM's test suite runs in the robot framework environment, which is a test automation
framework, and performs keyword-based test procedures. After that, it compares and analyses the response data
returned from the IUT with the expected data to determine the test result.
4.2.3 Test Environment
The Data Hub NGSI-LD interface conformance test equipment serves as a requester that sends a message to the Data
Hub under test according to the interface operation for each resource and the Data Hub operates as a responder. In
addition, test equipment plays the role of context producer that provides resources to the context broker of Data Hub
and context consumer that consumes resources.
ETSI
9 ETSI GR CIM 030 V1.1.1 (2023-12)

Figure 4.2.3-1: NGSI-LD interface test environment
4.2.4 Test case
For the Data Hub NGSI-LD interface test, figure 4.2.4-1 shows the test cases structure of ETSI GS CIM 012 [i.2]
interface test. The test cases were classified according to the common service functions and resources of the IUT.
Common service functions are divided into Provision and Consumption and each service function is divided into subtest
groups according to the target resource.

Figure 4.2.4-1: Test case structure for NGSI-LD interface
ETSI
10 ETSI GR CIM 030 V1.1.1 (2023-12)
4.3 Update history about NGSI-LD interface test code
4.3.1 Introduction
TTA and KETI cooperated in the development of a Data Hub through an R&D project of the Republic of Korea. The
role of TTA in this project was to develop a test system of the Data Hub's conformance to ETSI GS CIM 009 [i.1] and
conduct tests to ensure the reliability of the Data Hub developed by KETI. To fulfil its role, TTA developed a test
system as described above. As the purpose of developing the test system was to verify the conformance of the Data Hub
to ETSI GS CIM 009 [i.1], as explained above, the NGSI-LD functions applied by the Data Hub were confirmed in
advance to establish the test system.
Since the Data Hub did not apply all NGSI-LD functions, the test cases had to be designed according to the NGSI-LD
functions supported by the Data Hub. Then, the test code of the test cases was downloaded from ETSI CIM git, and the
code was validated to enhance the reliability of the test system. The Data Hub's NGSI-LD interface test system was
developed through this process. However, the validation had the following limitations. First, the test code of test suite
was a development version and not the official one. There are still some test cases which is not developed yet according
to ETSI GS CIM 013 [i.3]. Therefore, using an incomplete test suite was the only way to establish the test system of the
Data Hub.
In order to test the NGSI-LD interface of Data Hub, TTA developed the undeveloped test cases and updated the
incorrect ones which were not run according to the standard. Second, there was a time limit on completing the test cases
design for the NGSI-LD interface test. Due to the development speed of the Data Hub and the nature of R&D in Korea,
there was a deadline for establishing the test system. Instead of reviewing all of the test cases distributed by ETSI CIM,
the test cases for the functions supported by the Data Hub were selected, and TTA validated the test cases. Third, the
test cases were designed around the functions supported by the Data Hub because the purpose of establishing the test
system was to verify the conformance of the Data Hub. Therefore, TTA considered only the NGSI-LD function of the
Data Hub for the test system and did not validate the other test cases. Regardless of the limitations above, validation of
the NGSI-LD test code contributed to increasing the reliability of test suite.

Figure 4.3.1-1: Process for validation of NGSI-LD interface test equipment
4.3.2 Test cases Design
The test cases distributed by ETSI CIM consist of 277 cases. As described above, 76 test cases that can be used for the
test were selected among the NGSI-LD functions supported by the Data Hub.

Figure 4.3.2-1: Supported Test cases of NGSI-LD Test Suite
ETSI
11 ETSI GR CIM 030 V1.1.1 (2023-12)
4.3.3 Validation
It is necessary to install the Test Suite in the test equipment and check whether the test equipment operates correctly
based on the test standards. This series of verification processes is called validation, and the Data Hub was used to
validate the test equipment. Two validations are performed, and test standards are carefully reviewed to ensure that the
Data Hub and test equipment operate correctly according to the Standards. At the same time, the scope of test
verification is confirmed by checking whether the Data Hub supports the interface's operation for each test cases.

Figure 4.3.3-1: Validation Process of NGSI-LD Test Suite
4.3.4 Validation Result
As shown in table 4.3.4-1, a total of 41 detailed test cases passed the first validation, and a total of 73 test cases passed
the second validation. TTA conducted two NGSI-LD interface conformance tests on Data Hub using the test equipment.
Some of the test cases that failed in the first validation were corrected and supplemented. The correction and
supplementation history are described in clause 4.3.5. Secondary validation was conducted for the corrected and
supplemented test cases, and the validation result was derived based on the results. Through the validation process, it
was confirmed that the Data Hub supports interface operations for a total of 73 test cases, and the scope of the interface
conformance test was determined based on ETSI GS CIM 009 test framework using the validation results.
ETSI
12 ETSI GR CIM 030 V1.1.1 (2023-12)
Table 4.3.4-1: Validation Result
Validation Result
NGSI-LD API Case in
Test cases First Second Support for
Operation Detail
Validation Validation Data Hub
01 FAIL FAIL NOT SUPPORT
02 PASS PASS SUPPORT
TP/NGSI-LD/CI/Prov/E/001_01
03 PASS PASS SUPPORT
PASS PASS SUPPORT
01 FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/E/001_02 02 FAIL FAIL NOT SUPPORT
03 FAIL FAIL NOT SUPPORT
Create Entity
TP/NGSI-LD/CI/Prov/E/001_03 - PASS PASS SUPPORT
FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/E/001_04 -
TP/NGSI-LD/CI/Prov/E/001_05 - FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/E/001_06 - FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/E/001_07 - FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/E/001_08 - PASS PASS SUPPORT
FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/E/001_09 -
TP/NGSI-LD/CI/Prov/E/002_01 - PASS PASS SUPPORT
01 FAIL FAIL NOT SUPPORT
Delete Entity TP/NGSI-LD/CI/Prov/E/002_02
02 PASS PASS SUPPORT
TP/NGSI-LD/CI/Prov/E/002_03 - FAIL PASS SUPPORT
01 FAIL FAIL NOT SUPPORT
FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/BE/003_01
FAIL PASS SUPPORT
04 FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/BE/003_02 - FAIL FAIL NOT SUPPORT
01 FAIL FAIL NOT SUPPORT
Create batch of TP/NGSI-LD/CI/Prov/BE/003_03
02 FAIL FAIL NOT SUPPORT
Entities
TP/NGSI-LD/CI/Prov/BE/003_04 - FAIL PASS SUPPORT
FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/BE/003_05 -
TP/NGSI-LD/CI/Prov/BE/003_06 - FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/BE/003_07 - FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/BE/003_08 - FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/BE/003_09 - FAIL FAIL NOT SUPPORT
01 FAIL PASS SUPPORT
FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/BE/004_01 02
FAIL PASS SUPPORT
01 FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/BE/004_02 02 FAIL PASS SUPPORT
03 FAIL PASS SUPPORT
01 PASS PASS SUPPORT
Upsert batch of
TP/NGSI-LD/CI/Prov/BE/004_03 02 PASS PASS SUPPORT
Entities
PASS PASS SUPPORT
FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/BE/004_04 02 FAIL PASS SUPPORT
03 FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/BE/004_05 - FAIL FAIL NOT SUPPORT
01 FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/BE/004_06
FAIL FAIL NOT SUPPORT
PASS PASS SUPPORT
TP/NGSI-LD/CI/Prov/BE/005_01 02 PASS PASS SUPPORT
03 FAIL FAIL NOT SUPPORT
01 PASS PASS SUPPORT
Update batch of
TP/NGSI-LD/CI/Prov/BE/005_02 02 PASS PASS SUPPORT
Entities
03 FAIL FAIL NOT SUPPORT
PASS PASS SUPPORT
TP/NGSI-LD/CI/Prov/BE/005_03 -
FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/BE/005_04
02 FAIL FAIL NOT SUPPORT
ETSI
13 ETSI GR CIM 030 V1.1.1 (2023-12)
Validation Result
NGSI-LD API
Case in
Test cases First Second Support for
Operation Detail
Validation Validation Data Hub
TP/NGSI-LD/CI/Prov/BE/006_01 - PASS PASS SUPPORT
Delete batch of TP/NGSI-LD/CI/Prov/BE/006_02 - FAIL PASS SUPPORT
Entities 01 FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/BE/006_03
02 FAIL FAIL NOT SUPPORT
PASS PASS SUPPORT
TP/NGSI-LD/CI/Prov/EA/010_01
FAIL FAIL NOT SUPPORT
01 FAIL FAIL NOT SUPPORT
Append Entity
Attributes TP/NGSI-LD/CI/Prov/EA/010_02 02 PASS PASS SUPPORT
03 FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/EA/010_03 - FAIL PASS SUPPORT
01 PASS PASS SUPPORT
TP/NGSI-LD/CI/Prov/EA/011_01
FAIL FAIL NOT SUPPORT
FAIL FAIL NOT SUPPORT
Update Entity 01
Attributes TP/NGSI-LD/CI/Prov/EA/011_02 02 PASS PASS SUPPORT
03 FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/EA/011_03 - FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/EA/012_01 01 PASS PASS SUPPORT
01 FAIL FAIL NOT SUPPORT
PASS PASS SUPPORT
03 PASS PASS SUPPORT
Partial Update Entity TP/NGSI-LD/CI/Prov/EA/012_02
04 PASS FAIL NOT SUPPORT
Attributes
05 PASS PASS SUPPORT
06 PASS PASS SUPPORT
01 FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/EA/012_03
FAIL FAIL NOT SUPPORT
FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Prov/EA/013_01
03 FAIL FAIL NOT SUPPORT
01 FAIL FAIL NOT SUPPORT
Delete Entity Attributes TP/NGSI-LD/CI/Prov/EA/013_02 02 PASS PASS SUPPORT
03 FAIL FAIL NOT SUPPORT
01 FAIL PASS SUPPORT
TP/NGSI-LD/CI/Prov/EA/013_03
FAIL FAIL NOT SUPPORT
PASS PASS SUPPORT
TP/NGSI-LD/CI/Cons/E/018_01 02 PASS PASS SUPPORT
03 PASS PASS SUPPORT
01 FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Cons/E/018_02
02 FAIL PASS SUPPORT
Retrieve Entity
01 PASS PASS SUPPORT
TP/NGSI-LD/CI/Cons/E/018_03
PASS PASS SUPPORT
TP/NGSI-LD/CI/Cons/E/018_04 - PASS PASS SUPPORT
01 PASS PASS SUPPORT
TP/NGSI-LD/CI/Cons/E/018_06
02 PASS PASS SUPPORT
01 FAIL FAIL NOT SUPPORT
02 FAIL FAIL NOT SUPPORT
FAIL FAIL NOT SUPPORT
TP/NGSI-LD/CI/Cons/E/019_01 03
PASS PASS SUPPORT
05 FAIL FAIL NOT SUPPORT
01 FAIL PASS SUPPORT
Query Entities
02 PASS PASS SUPPORT
TP/NGSI-LD/CI/Cons/E/019_03 03 FAIL FAIL NOT SUPPORT
04 FAIL PASS SUPPORT
PASS PASS SUPPORT
TP/NGSI-LD/CI/Cons/E/019_04 - PASS PASS SUPPORT
TP/NGSI-LD/CI/Cons/E/019_06 - FAIL FAIL NOT SUPPORT
Retrieve Available TP/NGSI- 01 PASS PASS SUPPORT
Entity Types LD/CI/CONS/DISC/022_01 02 PASS PASS SUPPORT
Retrieve Details of TP/NGSI- 01 FAIL PASS SUPPORT
FAIL PASS SUPPORT
Available Entity Types LD/CI/CONS/DISC/023_01 02
FAIL PASS SUPPORT
Create Subscription TP/NGSI-LD/CI/SUB/028_03 -
Update Subscription TP/NGSI-LD/CI/SUB/029_05 - FAIL PASS SUPPORT
Retrieve Subscription TP/NGSI-LD/CI/SUB/030_03 - FAIL PASS SUPPORT
Delete Subscription TP/NGSI-LD/CI/SUB/032_03 - FAIL PASS SUPPORT
ETSI
14 ETSI GR CIM 030 V1.1.1 (2023-12)
4.3.5 Update History
4.3.5.0 Introduction
As a result of installing the Test Suite in the test equipment and conducting validation, several FAIL issues occurred.
Generally, four issues were found, for which updates were performed. The first issue is that the test procedure of the test
case covers a wider scope than the test purpose. In other words, although the purpose of the test was achieved in the test
procedure, some test cases were failed because they have additional test procedure unrelated to the test purpose. In the
test code, there are two Retrieve requests, one is Retrieve with context and the other is Retrieve without context. The
test equipment always determines failure because the IUT's response is either with or without context, so it is not
possible to satisfy both conditions. To solve this problem, the issued test cases were modified by deleting the additional
Retrieve request without context in the test suite code. Secondly, it was found that some test procedures did not adhere
to the technical specifications and consequently failed to obtain the expected test results. According to ETSI
GS CIM 009 [i.1], datasetid is not mandatory when retrieving for entities. However, but test suite was developed to
make sure the datasetid is a match when retrieving for entities, making it operate differently from the actual test
purpose. As the result, the part that checks the datasetid was deleted. Third, the source code of the test suite was not
developed according to the Test Standards. The test was not performed according to the intended test procedure, and it
was modified according to the test standard to ensure proper operation.
4.3.5.1 Deletion of Unnecessary Retrieve Request for Test Purpose
Table 4.3.5.1-1: Update history about deletion of Retrieve Request
TP/NGSI-LD/CI/Prov/E/001_04
Target Test TP/NGSI-LD/CI/Prov/E/001_07
case TP/NGSI-LD/CI/Prov/BE/003_04
TP/NGSI-LD/CI/Prov/BE/003_07
There were test cases where the test case failed the test due to additional test procedures even
Issues
though it fulfilled the test goal and passed.
Details of
Delete the additional Retrieve request without context in the Test Suite code.
Modification
4.3.5.2 datasetid
Table 4.3.5.2-1: Update history about datasetid
TP/NGSI-LD/CI/Prov/EA/010_01
TP/NGSI-LD/CI/Prov/EA/010_02
TP/NGSI-LD/CI/Prov/EA/011_01
Target Test
TP/NGSI-LD/CI/Prov/EA/012_01
case
TP/NGSI-LD/CI/Prov/EA/012_02
TP/NGSI-LD/CI/Prov/EA/013_02
TP/NGSI-LD/CI/Prov/EA/013_03
The test equipment retrieved for datasetid although datasetid is not essential regarding technical
Issues
standards.
Details of Remove the part where the test equipment requires to retrieve datasetid so that the datasetid does
Modification not affect to the test verdict.

4.3.5.3 Mismatch between Test Standards and Test equipment
Table 4.3.5.3-1: Update history about mismatch between standard and equipment
Target Test
TP/NGSI-LD/CI/SUB/028_03
case
Test standards and the source code developed based on them are different from the results
Issues
expected from test results. Modified the source code based on test standards.
Details of
Modified "Check Response Status Code Set to 200".
Modification
ETSI
15 ETSI GR CIM 030 V1.1.1 (2023-12)
5 NGSI-LD based Data Model Test Validation
5.1 Introduction
This clause explains how to test and validate the data model conformance of the Data Hub using NGSI-LD. A data
model represents the structure of data stored and managed in the Data Hub. It is necessary to use a unified data model to
ensure data interoperability between the various smart city platforms. Even when the same data model is used, the data
platform might generate data according to its own way. In order to test the ability of generating data according to the
data model, the data model conformance test was developed using the NGSI-LD interface.
5.2 Test Environment
5.2.1 Implementation Under Test
The data model stored and managed based on the Data Hub schema is defined as a test target, and the data model can be
exchanged with internal and external systems through the Central Broker supporting NGSI-LD-based interface.
5.2.2 Test Equipment
The test equipment is a Robot Framework-based test equipment verification software. It drives the validation module
based on user management for the IUT, System Under Test (SUT) management, test cases management, test execution
control, and test report management functions. The verification module interprets the test information, executes pre-
processing, test execution, and post-processing according to the test scenario, and judges the test results. The
configuration of the test equipment system is as shown in figure 5.2.2-1.

Figure 5.2.2-1: System architecture of Data model test equipment
ETSI
16 ETSI GR CIM 030 V1.1.1 (2023-12)
The test equipment that tests the conformance of the data model uses the NGSI-LD interface to send data Create and
Retrieve requests to the IUT according to the data model and confirms whether the data is created according to the data
model. For this, the schema file of the data model and the function to manage the file are required to proceed with the
test. Also, it is necessary to create and manage test data to test the data model. The data operate according to the test
procedure through the test tool and library in the robot framework, and the test equipment automatically creates a test
report based on the test result and log.

Figure 5.2.2-2: Process of Data Model test equipment
The operation of the test equipment provides user management for standard conformance test, System Under Test
(SUT) management, data model schema registration, test cases management, test execution control, and test report
management functions to utilize the test environment for the module to be tested and supports test related operations
ranging from the test execution and the preparation of the test result report. The conformance test workflow of the test
equipment is as shown in figure 5.2.2-3.

Figure 5.2.2-3: Decision process of Data model test equipment
ETSI
17 ETSI GR CIM 030 V1.1.1 (2023-12)
The test equipment performs the process shown in figure 5.2.2-4 to test the data model. First, the JSON Schema file of
the data model is uploaded to the test equipment to determine whether or not there is a syntax error in the JSON Schema
file. Then, if there are no syntax errors, data for expectation data and sending the creation request are created through
Test Data Generation. Expectation data serves as a list of correct answers to check whether the test equipment created
the data model as specified in the schema by retrieving for entities created by the creation request sent to the IUT.
Request data is created using the JSON schema file of the data model, and the test equipment sends a request message
to the IUT to create an entity. Then, the test equipment transmits a message to retrieve the entity created by the IUT.
The IUT compares the previously created expectation data with the entity that was retrieve d to ensure that the IUT
created the entity according to the JSON Schema of the data model.

Figure 5.2.2-4: Test process of Data model equipment
5.2.3 Test Environment
The data model conformance test equipment supports storing and managing the data model of the IUT based on the
schema. It also includes the Context Producer function to create and update data and the Context Consumer function to
inquire about and subscribe to data. Through this function, the data model of the smart city Data Hub under test can be
exchanged through the Central Broker. The data model Data Hub conformance test verifies whether the IUT transmits a
response message that conforms to technical specifications for the request message from the test equipment as shown in
figure 5.2.3-1.
Figure 5.2.3-1: Test environment of Data model test
5.3 Test case
The test cases list is classified into a valid behaviour test and a behaviour test. The valid behaviour test consists of test
cases for verifying the normal behaviour of the supported interface of the IUT, and the invalid behaviour test consists of
test cases used to request an interface behaviour not supported by the IUT and to verify whether errors are handled for
the request. In addition, reference identifiers are defined so that the interface defined in ETSI GS CIM 009 [i.1] can be
referred to for each test cases. Tables 5.3.1-1 and 5.3.2-1 show the valid operation test cases list and the invalid
operation test cases list, respectively.
ETSI
18 ETSI GR CIM 030 V1.1.1 (2023-12)
5.3.1 Valid Behaviour (BV)
Table 5.3.1-1: Valid behaviour about Data Model test
Test Set Operation Test Purpose Test cases
Checks whether a response message 2001
(CREATED) is sent when the creation of an entity TC/SCDH/COR/RES/CRE/BV-01
Create
is requested.
Resource
Checks whether a response message 2001
(CREATE)
(CREATED) is sent when the creation of an entity TC/SCDH/COR/RES/CRE/BV-02
is requested.
Checks whether a response message 2000 (OK)
and corresponding data are sent when an inquiry TC/SCDH/COR/RES/RET/BV-01
with an entity type is requested.
Checks whether a response message 2000 (OK)
and corresponding data are sent when an inquiry TC/SCDH/COR/RES/RET/BV-02
of a specific entity is requested.
Checks whether a response message 2000 (OK)
and corresponding data are sent when an inquiry is
TC/SCDH/COR/RES/RET/BV-03
requested with the attribute name of a specific
Retrieve
entity.
Resource
Checks whether a response message 2000 (OK)
(RETRIEVE)
and corresponding data are sent when an inquiry TC/SCDH/COR/RES/RET/BV-04
Data Core of history data with the entity type is requested.
Module Checks whether a response message 2000 (OK)
and corresponding data are sent when an inquiry TC/SCDH/COR/RES/RET/BV-05
of history data of a specific entity is requested.
Checks whether a response message 2000 (OK)
and corresponding data are sent when an inquiry
TC/SCDH/COR/RES/RET/BV-06
of history data matching a specific entity's time is
requested.
Checks whether a response message 2004
(CHANGED) is sent when the update of all TC/SCDH/COR/RES/UPD/BV-01
Update
attributes of a specific entity is requested.
Resource
Checks whether a response message 2004
(UPDATE)
(CHANGED) is sent when the update of a single TC/SCDH/COR/RES/UPD/BV-02
attribute of a specific entity is requested.
Checks whether a response message 2002
(DELETED) is sent when the deletion of a specific TC/SCDH/COR/RES/DEL/BV-01
Delete
entity is requested.
Resource
Checks whether a response message 2002
(DELETE)
(DELETED) is sent when the deletion of a single TC/SCDH/COR/RES/DEL/BV-02
attribute of a specific entity is requested.

ETSI
19 ETSI GR CIM 030 V1.1.1 (2023-12)
5.3.2 Invalid Behaviour (BI)
Table 5.3.2-1: Invalid behaviour about Data Model test
Test Set Operation Test Purpose Test cases
Checks whether an error response message 4100
(ALREADY EXISTS) is sent when the creation of TC/SCDH/COR/RES/CRE/BI-01
an entity is requested redundantly.
Checks whether an error response message 4101
(MANDATORY PARAMETER MISSING) is sent
TC/SCDH/COR/RES/CRE/BI-02
when the creation of an entity is requested without
an essential parameter (ID).
Checks whether an error response message 4102
(INVALID PARAMETER TYPE) is sent when the
TC/SCDH/COR/RES/CRE/BI-03
creation of an entity is requested with a wrong
parameter type.
Checks whether an error response message 4004
(NOT FOUND) is sent when th
...