ETSI TS 118 115 V2.0.0 (2016-09)

oneM2M; Testing Framework (oneM2M TS-0015 version 2.0.0 Release 2)

DTS/oneM2M-000015

General Information

• Status: Published
• Publication Date: 12-Sep-2016
• Technical Committee: oneM2M - oneM2M
• Current Stage: 12 - Completion
• Due Date: 03-Oct-2016
• Completion Date: 13-Sep-2016

Standards Content (Sample)

ETSI TS 118 115 V2.0.0 (2016-09)






TECHNICAL SPECIFICATION
oneM2M;
Testing Framework
(oneM2M TS-0015 version 2.0.0 Release 2)

---------------------- Page: 1 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 2 ETSI TS 118 115 V2.0.0 (2016-09)



Reference
DTS/oneM2M-000015
Keywords
IoT, M2M, methodology, testing

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2016.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 3 ETSI TS 118 115 V2.0.0 (2016-09)
Contents
Intellectual Property Rights . 5
Foreword . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions and abbreviations . 6
3.1 Definitions . 6
3.2 Abbreviations . 8
4 Conventions . 8
5 Introduction to the oneM2M testing methodology . 8
6 Conformance testing. 10
6.1 Introduction . 10
6.2 Test architecture . 11
6.2.1 Selection of Implementation Under Test . 11
6.2.1.1 Definition . 11
6.2.1.2 oneM2M Service Layer Communication . 12
6.2.2 Identification of the Reference Points . 12
6.3 Development of Conformance Test Specifications . 12
6.3.1 Implementation Conformance Statement (ICS) . 12
6.3.2 Test Suite Structure & Test Purposes (TSS&TP) . 13
6.3.2.1 Introduction . 13
6.3.2.2 Test Suite Structure . 13
6.3.2.3 Test Purpose . 14
6.3.2.3.1 Introduction . 14
6.3.2.3.2 TP identifier . 15
6.3.2.3.3 Test objective . 16
6.3.2.3.4 Reference . 16
6.3.2.3.5 ICS selection . 16
6.3.2.3.6 TP behaviour . 16
6.3.3 Abstract Test Suite (ATS) . 18
6.3.3.1 Abstract protocol tester . 18
6.3.3.2 TTCN-3 test architecture . 21
6.3.3.3 Test configurations . 22
6.3.3.4 ATS conventions . 22
6.3.3.4.1 Importing XSD definition . 22
6.3.3.4.2 The TTCN-3 naming conventions . 22
6.3.3.5 Verification of TTCN-3 . 23
6.3.4 Implementation eXtra Information for Testing (IXIT) . 23
7 Interoperability testing . 24
7.1 Introduction . 24
7.2 Basic Concepts . 25
7.2.1 Overview . 25
7.2.2 System Under Test (SUT) . 25
7.2.2.0 Introduction . 25
7.2.2.1 Devices Under Test (DUT) . 26
7.2.2.2 Test interfaces . 26
7.2.3 Test Environment . 26
7.2.3.0 Introduction . 26
7.2.3.1 Test Descriptions . 26
7.2.3.2 Test drivers . 26
7.3 Development of Interoperability Test Specifications . 27
7.3.1 Overview . 27
ETSI

---------------------- Page: 3 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 4 ETSI TS 118 115 V2.0.0 (2016-09)
7.3.2 Generic SUT Architecture . 27
7.3.3 Test scenarios . 28
7.3.4 Test bed architecture and Interfaces. 28
7.3.5 Interoperable Functions Statement (IFS) . 29
7.3.6 Test Descriptions (TD) . 29
Annex A (informative): Example of ICS table . 32
A.1 Capability Statement . 32
History . 33

ETSI

---------------------- Page: 4 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 5 ETSI TS 118 115 V2.0.0 (2016-09)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Technical Specification (TS) has been produced by ETSI Partnership Project oneM2M (oneM2M).
ETSI

---------------------- Page: 5 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 6 ETSI TS 118 115 V2.0.0 (2016-09)
1 Scope
The present document specifies a testing framework defining a methodology for development of conformance and
interoperability test strategies, test systems and the resulting test specifications for oneM2M standards.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference/.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI TS 118 001: "oneM2M; Functional Architecture (oneM2M TS-0001)".
[2] ETSI TS 118 004: "Service layer Core Protocol (oneM2M TS-0004)".
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] oneM2M Drafting Rules.
NOTE: Available at http://www.onem2m.org/images/files/oneM2M-Drafting-Rules.pdf.
[i.2] ISO/IEC 9646 (all parts): "Information technology - Open Systems Interconnection - Conformance
testing methodology and framework".
[i.3] ETSI EG 202 237: "Methods for Testing and Specification (MTS); Internet Protocol Testing
(IPT); Generic approach to interoperability testing".
[i.4] ETSI ES 201 873-1: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; Part 1: TTCN-3 Core Language".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
conformance: compliance with requirements specified in applicable standards ISO/IEC 9646 [i.2]
ETSI

---------------------- Page: 6 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 7 ETSI TS 118 115 V2.0.0 (2016-09)
conformance testing: process for testing that an implementation is compliant with a protocol standard, which is
realized by test systems simulating the protocol with test scripts executed against the implementation under test
Device Under Test (DUT): combination of software and/or hardware items which implement the functionality of
standards and interact with other DUTs via one or more reference points
ICS proforma: document, in the form of a questionnaire, which when completed for an implementation or system
becomes an ICS
Implementation Conformance Statement (ICS): statement made by the supplier of an implementation or system
claimed to conform to a given specification, stating which capabilities have been implemented
Implementation eXtra Information for Testing (IXIT): checklist which contains or references all of the information
(in addition to that given in the ICS) related to the IUT and its testing environment, which will enable the test laboratory
to run an appropriate test suite against the IUT
Implementation Under Test (IUT): implementation of one or more Open Systems Interconnection (OSI) protocols in
an adjacent user/provider relationship, being the part of a real open system which is to be studied by testing
(ISO/IEC 9646-1 [i.2])
Inopportune Behaviour (BO): test group that handles invalid exchanges of messages, which are properly structured
and correctly encoded
interoperability: ability of two systems to interoperate using the same communication protocol
interoperability testing: activity of proving that end-to-end functionality between (at least) two devices is as required
by the base standard(s) on which those devices are based
InterWorking Function (IWF): translation of one protocol into another one so that two systems using two different
communication protocols are able to interoperate
Invalid Behaviour (BI): test group that handles valid exchanges of messages, which are either not properly structured
or incorrectly encoded
IXIT proforma: document, in the form of a questionnaire, which when completed for an implementation or system,
becomes an IXIT
Qualified Equipment (QE): grouping of one or more devices that has been shown and certified, by rigorous and
well-defined testing, to interoperate with other equipment
NOTE 1: Once an DUT has been successfully tested against a QE, it may be considered to be a QE, itself.
NOTE 2: Once a QE is modified, it loses its status as QE and becomes again an DUT.
test case: specification of the actions required to achieve a specific test purpose, starting in a stable testing state, ending
in a stable testing state and defined in either natural language for manual operation or in a machine-readable language
(such as TTCN-3) for automatic execution
testing framework: document providing guidance and examples necessary for the development and implementation of
a test specification
test purpose: description of a well-defined objective of testing, focussing on a single requirement or a set of related
requirements
Valid Behaviour (BV): test group that handles valid exchanges of messages, which are properly structured and
correctly
ETSI

---------------------- Page: 7 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 8 ETSI TS 118 115 V2.0.0 (2016-09)
3.2 Abbreviations
For the purposes of the present document, the terms and definitions given in ETSI TS 118 001 [1] and the following
apply:
API Application Programming Interface
APT Abstract Protocol Tester
ATS Abstract Test Suite
BI Invalid Behaviour
BO Inopportune Behaviour
BV Valid Behaviour
CoAP Constrained Application Protocol
EUT Equipment Under Test
FQDN Fully Qualified Domain Name
HTTP HyperText Transfer Protocol
IFS Interoperable Features Statement
IOP Interoperability
IUT Implementation Under Test
IWF InterWorking Function
JSON JavaScript Object Notation
MMI Man-Machine Interface
MQTT Message Queue Telemetry Transport
PICS Protocol Implementation Conformance Statement
QE Qualified Equipment
SUT System Under Test
TC Test Case
TCP Transmission Control Protocol
TD Test Description
TP Test Purpose
TSS Test Suite Structure
TTCN-3 Testing and Test Control Notation version 3
UDP User Datagram Protocol
URI Uniform Resource Identifier
XML eXtensible Markup Language
4 Conventions
The key words "Shall", "Shall not", "May", "Need not", "Should", "Should not" in the present document are to be
interpreted as described in the oneM2M Drafting Rules [i.1].
5 Introduction to the oneM2M testing methodology
The present document provides:
• Identification of the implementations under test (IUT) for conformance testing and the device under test
(DUTs) for interoperability, i.e. answering the question "what is to be tested".
• Definition of the applicable test procedures, i.e. answering the question "how is it to be tested".
• Definition of the procedure for development of test specifications and deliverables (for instance: TSS&TP, TP
proforma, TTCN-3 test suite and documentation).
Figure 1 illustrates the oneM2M testing framework and the interactions with oneM2M base standards and test
specifications. The oneM2M testing framework is based on concepts defined in ISO/IEC 9646 [i.2], TTCN-3 [i.4],
ETSI EG 202 237 [i.3].
ETSI

---------------------- Page: 8 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 9 ETSI TS 118 115 V2.0.0 (2016-09)
Base specifications
Conformance oneM2M Test Interoperability
Test Methodology Test
Specifications Specifications
Identification
IUTs of IUTs and DUTs
DUTs
PICS
IFS
Development
of test
specifications
TSS&TP
TDs
ATS IOP Test
Abstract Test
Bed
Method
TTCN-3

Figure 5-1: oneM2M testing methodology interactions
The test specifications are usually developed for a single base protocol standard or for a coherent set of standards. As
such, it is possible to follow the methodology specified for conformance test development in ISO/IEC 9646-1 [i.2]
without much difficulty. However, oneM2M testing requirements are, in many cases, distributed across a wide range of
documents and, thus, an adaptation of the ISO/IEC 9646 [i.2] approach to test development is necessary. Also, for
readability, consistency and to ease reusability of TTCN-3 code it is necessary to apply some guidelines on the use of
TTCN-3.
It is this approach that is referred to as the "oneM2M testing framework".
As its name implies, the framework is oriented towards the production of Test specifications. The oneM2M testing
Framework comprises:
• a documentation structure:
- catalogue of capabilities/features/functions (PICS or IFS);
- Test Suite Structure (TSS);
- Test Purposes:
Conformance;
Interoperability.
• a methodology linking the individual elements of a test specification together:
- style guidelines and examples;
- naming conventions;
- a structured notation for TP;
- guidelines on the development of TTCN-3 Test Cases (TCs);
- guidelines on the use of tabulated English Test Descriptions (TDs).
ETSI

---------------------- Page: 9 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 10 ETSI TS 118 115 V2.0.0 (2016-09)
6 Conformance testing
6.1 Introduction
The clause 6 shows how to apply the oneM2M conformance testing methodology in order to properly produce oneM2M
conformance test specifications.
The Conformance testing can show that a product correctly implements a particular standardized protocol, that is, it
establishes whether or not the implementation under test meets the requirements specified for the protocol itself.
EXAMPLE: It will test protocol message contents and format as well as the permitted sequences of messages.
In that context, tests are performed at open standardized interfaces that are not (usually) accessible
to an end user, and executed by a dedicated test system that has full control of the system under
test and the ability to observe all incoming and out coming communications; the high degree of
control of the test system over the sequence and contents of the protocol messages allows to test
both valid and invalid behaviour.
Conformance
Implementation
Test System
Under Test

Figure 6.1-1: Conformance testing
Conformance test specifications should be produced following the methodology described in ISO/IEC 9646-1 [i.2]. In
summary, this methodology begins with the collation and categorization of the features and options to be tested into a
tabular form which is normally referred to as the " Implementation Conformance Statement" (ICS). All implemented
capabilities supported by the Implementation Under Test (IUT) are listed by the implementer in the ICS, so that the
tester knows which options have to be tested. This ensures that complete coverage is obtained.
The next step is to collect the requirements from the specification that is tested. For each requirement, one or more tests
should be identified and classified into a number of groups which will provide a structure to the overall test suite (TSS).
A brief Test Purpose (TP) should then be written for each identified test and this should make it clear what is to be
tested but not how this should be done. Although not described or mandated in ISO/IEC 9646-1 [i.2], in many situations
(particularly where the TPs are complex) it may be desirable to develop a Test Description (TD) for each TP. The TD
describes in plain language (often tabulated) the actions required to reach a verdict on whether an implementation
passes or fails the test. Finally, a detailed Test Case (TC) is written for each TP. In the interests of test automation, TCs
are usually combined into an Abstract Test Suite (ATS) using a specific testing language such as TTCN-3. The TCs in
the ATS are then "Verified" against a number of IUTs for correct operation according to some agreed procedures,
before being released for use by the industry. An Implementation eXtra Information for Test (IXIT) proforma
associated to the ATS, should be produced in supplement of the ICS document and Test Cases to help to execute
Protocol conformance testing using oneM2M dedicated test equipment.
In summary, the oneM2M Conformance Testing methodology consists of:
• Selection of Implementations Under Test (IUT).
• Identification of reference points.
• Development of test specifications, which includes:
- Development of "Implementation Conformance Statements" (ICS), if not already provided as part of the
base standard.
- Development of "Test Suite Structure and Test Purposes" (TSS&TP).
- Development of "Abstract Test Suite and Implementation eXtra Information for Test" (ATS&IXIT)
including:
Definition of the Abstract Protocol Tester (APT).
Definition of TTCN-3 test architecture.
ETSI

---------------------- Page: 10 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 11 ETSI TS 118 115 V2.0.0 (2016-09)
Development of TTCN-3 test suite, e.g. naming conventions, code documentation, test case
structure.
Verification of ATS (TTCN-3)
IXIT proforma.
6.2 Test architecture
6.2.1 Selection of Implementation Under Test
6.2.1.1 Definition
The "Implementation Under Test" (IUT) is a protocol implementation considered as an object for testing. This means
that the test process will focus on verifying the compliance of this protocol implementation (IUT) with requirements set
up in the related base standard. An IUT normally is implemented in a "System Under Test" (SUT). For testing, a SUT is
connected to a test system over at least a single interface. Such an interface is identified as "Reference Point" (RP) in
the present document. Further details on RPs are presented in clause 6.2.2.
NOTE: Other interfaces between the test system and the IUT may be used to control the behaviour of the IUT
during the test process.
Figure 6.2.1.1-1 shows a complete view of communication layer for oneM2M domain. Further details are presented in
the following clauses.
IUT
oneM2M
oneM2M
Security
oonneMeM2M2M
Management
Service Layer
HTTP/CoAP/MQTT
Network & Transport
WiFi/6LoWPAN/
Ethernet, Zigbee,. Security
Management
Access

Figure 6.2.1.1-1: Example of IUT in the oneM2M reference architecture
ETSI

---------------------- Page: 11 ----------------------
(oneM2M TS-0015 version 2.0.0 Release 2) 12 ETSI TS 118 115 V2.0.0 (2016-09)
6.2.1.2 oneM2M Service Layer Communication
Table 6.2.1.2-1 shows the IUTs for oneM2M reference architecture as defined in ETSI TS 118 001 [1].
Table 6.2.1.2-1: IUTs for oneM2M
IUT (node) Entities Interfaces Notes
ASN Application Entity (AE) Mca
Common Services Entity (CSE) Mca, Mcc, Mcn
ADN Application Entity (AE) Mca
MN Application Entity (AE) Mca
Common Services Entity (CSE) Mca, Mcc, Mcn
IN Application Entity (AE) Mca
Common Services Entity (CSE) Mca, Mcc, Mcn,
Mcc’, Mch
ASN/MN/IN Network Services Entity (NSE) Mcn

Table 6.2.1.2-1 needs to be amended in the following cases:
• A new node or entity is defined on the base specifications.
• A new interface is defined on the base specifications between any of the existing nodes or entities.
6.2.2 Identification of the Reference Points
This clause illustrates candidate reference points (RPs) where test systems can be connected in order to test
conformance of oneM2M protocols (IUTs) with oneM2M base standards.
Table 6.2.2-1: RPs for oneM2M
RP Identifier RP Type oneM2M node- oneM2M node- Network
entity entity
RP-oneM2M-1 Mca ASN-AE ASN-CSE
RP-oneM2M-2 Mca MN-AE MN-CSE
RP-oneM2M-3 Mca IN-AE IN-CSE
RP-oneM2M-4 Mca ADN-AE IN-CSE
RP-oneM2M-5 Mca ADN-AE MN-CSE
RP-oneM2M-6 Mcc ASN-CSE IN-CSE
RP-oneM2M-7 Mcc ASN-CSE MN-CSE
...