ETSI TS 103 268-1 V1.1.1 (2017-04)

SmartM2M; Smart Appliances Ontology and Communication Framework Testing; Part 1: Testing methodology

DTS/SmartM2M-103 268-1 SAP_tst

General Information

• Status: Published
• Publication Date: 06-Apr-2017
• Technical Committee: SmartM2M - Smart M2M
• Current Stage: 12 - Completion
• Due Date: 15-Apr-2017
• Completion Date: 07-Apr-2017

Standards Content (Sample)

ETSI TS 103 268-1 V1.1.1 (2017-04)






TECHNICAL SPECIFICATION
SmartM2M;
Smart Appliances Ontology and Communication
Framework Testing;
Part 1: Testing methodology

---------------------- Page: 1 ----------------------
2 ETSI TS 103 268-1 V1.1.1 (2017-04)



Reference
DTS/SmartM2M-103 268-1 SAP_tst
Keywords
data, data sharing, IoT, M2M, Model,
multi service testing, oneM2M, ontology, testing

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2017.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI TS 103 268-1 V1.1.1 (2017-04)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions and abbreviations . 7
3.1 Definitions . 7
3.2 Abbreviations . 8
4 Introduction to the SAP testing methodology . 8
5 Conformance testing. 10
5.1 Introduction . 10
5.2 Implementation Under Test . 11
5.2.1 Introduction. 11
5.2.2 Service Layer Communication . 12
5.2.2.1 oneM2M implementation . 12
5.2.2.2 SAP implementation . 12
5.3 Identification of the Reference points . 12
5.4 Development of Conformance Test Specifications . 12
5.4.1 Protocol Implementation Conformance Statements (PICS) . 12
5.4.2 Test Suite Structure & Test Purposes (TSS & TP) . 13
5.4.2.1 Introduction . 13
5.4.2.2 Test Suite Structure . 13
5.4.2.3 Test Purpose . 13
5.4.2.3.1 Introduction . 13
5.4.2.3.2 TP identifier . 15
5.4.2.3.3 Test objective . 15
5.4.2.3.4 Reference . 15
5.4.2.3.5 PICS selection . 16
5.4.2.3.6 TP behaviour . 16
5.4.3 Abstract Test Suite (ATS) . 20
5.4.3.1 Abstract protocol tester . 20
5.4.3.2 TTCN-3 test architecture . 21
5.4.3.2.1 Introduction . 21
5.4.3.2.2 Generic Conformance architecture . 22
5.4.3.2.3 Smart Appliance Conformance architecture . 23
5.4.3.3 TTCN-3 test suite . 24
5.4.3.3.1 TTCN-3 Test architecture . 24
5.4.3.3.2 Importing XSD definition . 24
5.4.3.3.3 The TTCN-3 naming conventions . 24
5.4.3.3.4 TTCN-3 code documentation . 26
5.4.3.3.5 Test cases structure . 26
5.4.4 Protocol Implementation eXtra Information for Testing (PIXIT). 27
6 Interoperability testing . 27
6.1 Introduction . 27
6.2 Basic concepts for interoperability testing . 28
6.2.1 Overview . 28
6.2.2 System Under Test (SUT) . 28
6.2.2.1 Introduction . 28
6.2.2.2 Devices Under Test (DUT) . 28
6.2.2.3 Test interfaces . 29
6.2.3 Test Environment . 29
ETSI

---------------------- Page: 3 ----------------------
4 ETSI TS 103 268-1 V1.1.1 (2017-04)
6.2.3.1 Introduction . 29
6.2.3.2 Test Descriptions . 29
6.2.3.3 Test drivers . 29
6.3 Development of Interoperability Test Specifications . 30
6.3.1 Overview . 30
6.3.2 Generic SUT Architecture . 30
6.3.3 Test architecture and Interfaces . 30
6.3.4 Interoperable Functions Statement (IFS) . 32
6.3.5 Test Descriptions (TD) . 32
Annex A (informative): Example of ICS table . 34
A.1 SAREF Device Class Statement . 34
History . 35


ETSI

---------------------- Page: 4 ----------------------
5 ETSI TS 103 268-1 V1.1.1 (2017-04)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Technical Specification (TS) has been produced by ETSI Technical Committee Smart Machine-to-Machine
communications (SmartM2M).
The present document is part 1 of a multi-part deliverable covering Conformance test specifications for Smart
Appliances Ontology and Communication Framework Testing as identified below:
Part 1: "Testing methodology";
Part 2: "Protocol Implementation Conformance Statement (PICS) pro forma";
Part 3: "Test Suite Structure and Test Purposes (TSS & TP)";
Part 4: "Abstract Test Suite (ATS) and Protocol Implementation eXtra Information for Testing (PIXIT)".
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI

---------------------- Page: 5 ----------------------
6 ETSI TS 103 268-1 V1.1.1 (2017-04)
1 Scope
The scope of the present document is to support Smart Appliance common ontology and communication framework
testing needs. It specifies a global methodology for testing for Smart Appliances, based oneM2M specifications. It
analyses the overall testing needs and identifies and defines the additional documentation required.
The testing framework proposed in the present document provides methodology for development of conformance and
interoperability test strategies, test systems and the resulting test specifications for SAP.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI TS 103 264: "SmartM2M; Smart Appliances; Reference Ontology and oneM2M Mapping".
[2] ETSI TS 103 267: "SmartM2M; Smart Appliances; Communication Framework".
[3] ETSI TS 118 101: "oneM2M; Functional Architecture (oneM2M TS-0001)".
[4] ETSI TS 118 104: "oneM2M; Service Layer Core Protocol Specification (oneM2M TS-0004)".
[5] ETSI TS 118 112: "oneM2M; Base Ontology (oneM2M TS-0012)".
[6] ETSI TS 118 108: "oneM2M; CoAP Protocol Binding (oneM2M TS-0008)".
[7] ETSI TS 118 109: "oneM2M; HTTP Protocol Binding (oneM2M TS-0009)".
[8] ETSI TS 118 110: "oneM2M; MQTT Protocol Binding (oneM2M TS-0010)".
[9] ETSI TS 118 120: "oneM2M; WebSocket Protocol Binding (oneM2M TS-0020)".
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI EG 202 798: "Intelligent Transport Systems (ITS); Testing; Framework for conformance and
interoperability testing".
[i.2] ISO/IEC 9646 (all parts): "Information technology - Open Systems Interconnection - Conformance
testing methodology and framework".
ETSI

---------------------- Page: 6 ----------------------
7 ETSI TS 103 268-1 V1.1.1 (2017-04)
[i.3] ETSI ES 201 873 (all parts): "Methods for Testing and Specification (MTS); The Testing and Test
Control Notation version 3".
[i.4] ETSI EG 202 237: "Methods for Testing and Specification (MTS); Internet Protocol Testing
(IPT); Generic approach to interoperability testing".
[i.5] ETSI EG 201 058: "Methods for Testing and Specification (MTS); Implementation Conformance
Statement (ICS) pro forma style guide".
[i.6] ETSI ETR 266: "Methods for Testing and Specification (MTS); Test Purpose style guide".
[i.7] ETSI ETS 300 406: "Methods for Testing and Specification (MTS); Protocol and profile
conformance testing specifications; Standardization methodology".
[i.8] ISO 10746 (all parts): "Information technology - Open Distributed Processing - Reference model".
[i.9] ETSI TS 103 268-2:"SmartM2M; Smart Appliances Ontology and Communication Framework
Testing; Part 2: Protocol Implementation Conformance Statement (PICS) proforma".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the terms and definitions given in ISO 10746 [i.8], ETSI TS 103 264 [1],
ETSI TS 103 267 [2], ETSI TS 118 101 [3], ETSI TS 118 104 [4] and the following apply:
conformance testing: process for testing that an implementation is compliant with a protocol standard, which is
realized by test systems simulating the protocol with test scripts executed against the implementation under test
interoperability testing: activity of proving that end-to-end functionality between (at least) two devices is as required
by the base standard(s) on which those devices are based
testing framework: document providing guidance and examples necessary for the development and implementation of
a test specification
conformance: compliance with requirements specified in applicable standards ISO/IEC 9646 [i.2]
Device Under Test (DUT): combination of software and/or hardware items which implement the functionality of
standards and interact with other DUTs via one or more reference points
Implementation Under Test (IUT): implementation of one or more Open Systems Interconnection (OSI) protocols in
an adjacent user/provider relationship, being the part of a real open system which is to be studied by testing
(ISO/IEC 9646-1 [i.2])
interoperability: ability of two systems to interoperate using the same communication protocol
interoperability test suite: collection of test cases designed to prove the ability of two (or more) systems to
interoperate
InterWorking Function (IWF): translation of one protocol into another one so that two systems using two different
communication protocols are able to interoperate
Qualified Equipment (QE): grouping of one or more devices that has been shown and certified, by rigorous and
well-defined testing, to interoperate with other equipment
NOTE 1: Once a DUT has been successfully tested against a QE, it may be considered to be a QE, itself.
NOTE 2: Once a QE is modified, it loses its status as QE and becomes again a DUT.
test case: specification of the actions required to achieve a specific test purpose, starting in a stable testing state, ending
in a stable testing state and defined in either natural language for manual operation or in a machine-readable language
(such as TTCN-3) for automatic execution
ETSI

---------------------- Page: 7 ----------------------
8 ETSI TS 103 268-1 V1.1.1 (2017-04)
test purpose: description of a well-defined objective of testing, focussing on a single interoperability requirement or a
set of related interoperability requirements
3.2 Abbreviations
For the purposes of the present document, the abbreviations given in ISO 10746 [i.8], ETSI TS 103 268-2 [i.9], ETSI
TS 118 101 [3], ETSI TS 118 104 [4] and the following apply:
API Application Programming Interface
ATS Abstract Test Suite
DUT Device Under Test
EUT Equipment Under Test
IFS Interoperable Features Statement
IUT Implementation Under Test
IWF InterWorking Function
PICS Protocol Implementation Conformance Statement
PIXIT Protocol Implementation eXtra Information for Testing
QE Qualified Equipment
RP Reference Point
SAP Smart Appliance
SUT System Under Test
TP Test Purpose
TSS Test Suite Structure
TST Testing
4 Introduction to the SAP testing methodology
The present document provides:
• Identification of the implementations under test (IUT) for conformance testing and the device under test
(DUTs) for interoperability, i.e. answering the question "what is to be tested".
• Definition of the applicable test procedures, i.e. answering the question "how is to be tested".
• Definition of the procedure for development of test specifications and deliverables (for instance: TSS & TP,
TP pro forma, TTCN-3 test suite and documentation).
Figure 4-1 illustrates the SAP M2M testing framework and the interactions with M2M base standards and SAP M2M
test specifications. The SAP M2M testing framework is based on concepts defined in ISO/IEC 9646 [i.2], TTCN-3 [i.3],
ETSI EG 202 237 [i.4] and ETSI EG 202 798 [i.1].
ETSI

---------------------- Page: 8 ----------------------
9 ETSI TS 103 268-1 V1.1.1 (2017-04)
Base specifications
Conformance SAP Test Interoperability
Test Test
Methodology
Specifications Specifications
Identification
IUTs of IUTs and DUTs
DUTs
PICS
IFS
Development
of test
specifications
TSS&TP
TDs
IOP Test
ATS
Abstract Test
Bed
Method
TTCN-3

Figure 4-1: SAP M2M testing methodology interactions
ETSI test specifications are usually developed for a single base protocol standard or for a coherent set of standards. As
such, it is possible to follow the methodology specified for conformance test development in ISO/IEC 9646-1 [i.2]
without much difficulty. However, M2M and Smart Appliance testing requirements are, in many cases, distributed
across a wide range of documents and, thus, an adaptation of the ISO/IEC 9646 [i.2] approach to test development is
necessary. Also, for readability, consistency and to ease reusability of TTCN-3 code it is necessary to apply some
guidelines on the use of TTCN-3.
It is this approach that is referred to as the "5Smart Appliance testing framework".
As its name implies, the framework is oriented towards the production of Test specifications. The Smart Appliance
testing Framework comprises:
• A documentation structure:
- Catalogue of requirements (PICS or IFS).
- Test Suite Structure (TSS).
- Test Purposes:
Conformance.
Interoperability.
• A methodology linking the individual elements of a test specification together:
- Style guidelines and examples.
- Naming conventions.
- A structured notation for TP.
- Guidelines on the development of TTCN-3 Test Cases (TCs).
- Guidelines on the use of tabulated English Test Descriptions (TDs).
ETSI

---------------------- Page: 9 ----------------------
10 ETSI TS 103 268-1 V1.1.1 (2017-04)
5 Conformance testing
5.1 Introduction
The following clauses show how to apply the ETSI conformance testing methodology to SAP M2M in order to properly
produce SAP M2M conformance test specifications.
The Conformance testing can show that a product correctly implements a particular standardized protocol, that is, it
establishes whether or not the implementation under test meets the requirements specified for the protocol itself.
For example, it will test protocol message contents and format as well as the permitted sequences of messages. In that
context, tests are performed at open standardized interfaces that are not (usually) accessible to an end user, and executed
by a dedicated test system that has full control of the system under test and the ability to observe all incoming and out
coming communications; the high degree of control of the test system over the sequence and contents of the protocol
messages allows to test both valid and invalid behaviour.
Conformance
Implementation
Test System
Under Test

Figure 5.1-1: Conformance testing
Conformance test specifications should be produced following the methodology described in ISO/IEC 9646-1 [i.2]. In
summary, this methodology begins with the collation and categorization of the requirements to be tested into a tabular
form which is normally referred to as the "Protocol Implementation Conformance Statement" (PICS). Each PICS relates
to a specific protocol standard. In those cases where the requirements are distributed across a large number of
documents there may be very little benefit in producing an individual PICS for each document. Consequently, the
requirements should be collected together and categorized in a single document, referred as the Requirements
Catalogue. The present document could be structured as an overall PICS covering the requirements of all the relevant
specifications.
For each requirement in the catalogue, one or more tests should be identified and classified into a number of groups
which will provide a structure to the overall test suite (TSS). A brief Test Purpose (TP) should then be written for each
identified test and this should make it clear what is to be tested but not how this should be done. Although not described
or mandated in ISO/IEC 9646-1 [i.2], in many situations (particularly where the TPs are complex) it may be desirable to
develop a Test Description (TD) for each TP. The TD describes in plain language (often tabulated) the actions required
to reach a verdict on whether an implementation passes or fails the test. Finally, a detailed Test Case (TC) is written for
each TP. In the interests of test automation, TCs are usually combined into an Abstract Test Suite (ATS) using a
specific testing language such as TTCN-3.
In summary, the SAP M2M Conformance Testing methodology consists of:
• Selection of Implementations Under Test (IUT).
• Identification of reference points.
• Development of test specifications, which includes:
- Development of "Implementation Conformance Statements" (ICS), if not already provided as part of the
base standard.
- Development of "Test Suite Structure and Test Purposes" (TSS & TP).
- Development of "Abstract Test Suite" (ATS) including:
Definition of the Abstract Protocol Tester (APT).
Definition of TTCN-3 test architecture.
Development of TTCN-3 test suite, e.g. naming conventions, code documentation, test case
structure.
ETSI

---------------------- Page: 10 ----------------------
11 ETSI TS 103 268-1 V1.1.1 (2017-04)
5.2 Implementation Under Test
5.2.1 Introduction
The "Implementation Under Test" (IUT) is a protocol implementation considered as an object for testing. This means
that the test process will focus on verifying the compliance of this protocol implementation (IUT) with requirements set
up in the related base standard. An IUT normally is implemented in a "System Under Test" (SUT). For testing, an SUT
is connected to a test system over at least a single interface. Such an interface is identified as "Reference Point" (RP) in
the present document. Further details on RPs are presented in clause 5.2.
NOTE: Other interfaces between the test system and the IUT may be used to control the behaviour of the IUT
during the test process.
IUTs normally are entities of a protocol architecture for a specific communication protocol located in an OSI layer.
Figure 5.2.1-1 shows a complete view of communication layer for M2M domain. Further details are presented in the
clause 5.2.2.
IUT
oneM2M
oneM2M
Security
ononeeM2MM2M
Management
Service Layer
HTTP/CoAP/MQTT
Network & Transport
WiFi/6LoWPAN/
Ethernet, Zigbee,.
Security
Management
Access

Figure 5.2.1-1: Example of IUT in the oneM2M reference architecture
ETSI

---------------------- Page: 11 ----------------------
12 ETSI TS 103 268-1 V1.1.1 (2017-04)
5.2.2 Service Layer Communication
5.2.2.1 oneM2M implementation
Table 5.2.2.1-1 shows the IUTs for oneM2M reference architecture as defined in ETSI TS 118 101 [3].
Table 5.2.2.1-1: IUTs for oneM2M
IUT (node) Entities Interfaces Notes
ASN Application Entity (AE) Mca
Common Services E
...