SIST-V ETSI/EG 202 237 V1.2.1:2016
(Main)Methods for Testing and Specification (MTS) - Internet Protocol Testing (IPT) - Generic approach to interoperability testing
Methods for Testing and Specification (MTS) - Internet Protocol Testing (IPT) - Generic approach to interoperability testing
This document gives general guidance on the specification and execution of interoperability tests for communication systems specifically in the context of product certification. It provides a framework within which interoperability test specifications for a wide range of product types can be developed. The guidelines are expressed as recommendations rather than strict rules and leave enough freedom to allow test specifiers to adopt and adapt processes to suit each particular project while still ensuring that test specifications accurately reflect the requirements of the base standards and can be executed consistently across a range of configurations.
Metode za preskušanje in specificiranje (MTS) - Preskušanje internetnega protokola - Generični pristop k preskušanju medobratovalnosti
Ta dokument podaja splošne smernice glede specifikacije in izvedbe preskusov medobratovalnosti za komunikacijske sisteme zlasti v kontekstu certificiranja proizvodov. Zagotavlja okvir za možnost razvoja specifikacij preskusov medobratovalnosti za številne vrste proizvodov. Smernice so izražene kot priporočila, ne kot stroga pravila, ter puščajo dovolj svobode, da lahko izdajatelji specifikacij preskusov sprejmejo in priredijo postopke tako, da ti ustrezajo posameznemu projektu, obenem pa zagotovijo, da specifikacije preskusov natančno odražajo zahteve osnovnih standardov in jih je mogoče dosledno izvajati v različnih konfiguracijah.
General Information
Standards Content (Sample)
Final draft ETSI EG 202 237 V1.2.1 (2010-06)
ETSI Guide
Methods for Testing and Specification (MTS);
Internet Protocol Testing (IPT);
Generic approach to interoperability testing
2 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
Reference
REG/MTS-00129-IOP-TstApproach
Keywords
interoperability, IP, telephony, testing
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2010.
All rights reserved.
TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
Contents
Intellectual Property Rights . 6
Foreword . 6
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definitions and abbreviations . 8
3.1 Definitions . 8
3.2 Abbreviations . 9
4 Types of testing . 9
4.1 Interoperability testing . 9
4.2 Conformance testing. 10
4.3 Combining interoperability testing and conformance testing . 11
5 Interoperability testing process overview . 11
6 Basic concepts . 12
6.1 Means of Testing . 12
6.2 Equipment Under Test (EUT) . 13
6.3 Qualified Equipment (QE) . 13
6.3.1 QEs and Devices . 13
6.3.2 Designating the first QE . 13
6.4 System Under Test (SUT) . 13
6.5 Test interface . 14
6.6 Test driver . 14
6.7 Test coordinator . 14
6.8 Interoperability test cases . 14
6.9 Means of Communication . 15
7 Generic interoperability test architectures . 15
7.1 Test architectures with a single QE . 15
7.2 Test architectures with multiple QEs. 16
7.2.1 An example using 3 QEs . 16
7.2.2 Testing IP hosts with multiple QEs . 17
8 Developing interoperability tests . 18
8.1 Overview . 18
8.2 Specify abstract architecture. 19
8.3 Prepare draft IFS Proforma . 19
8.4 Specify Test Suite Structure . 20
8.4.1 Identify test groups . 20
8.4.2 Define test coverage within each test group . 20
8.5 Write Test Purposes. 21
8.6 Write test cases . 21
8.6.1 Pre-test conditions . 21
8.6.2 Test steps and verdicts . 22
8.6.2.1 Test steps . 22
8.6.2.2 Verdicts . 22
8.6.2.3 Specification of test steps and verdicts. 22
8.6.3 Example . 23
8.6.4 Pre-amble and post-amble . 24
8.6.4.1 Alternative test case presentation forms . 25
8.7 Validate test cases . 26
8.8 Finalize IFS . 26
9 Interoperability testing process . 26
ETSI
4 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
9.1 Overview . 26
9.2 Prepare for testing . 27
9.2.1 Test arrangement . 27
9.2.2 Test planning . 28
9.3 Testing . 29
9.3.1 Manual testing . 29
9.3.2 Automated testing . 29
9.4 Write test report . 29
Annex A: Example IFS (Internet Key Exchange protocol, IKEv2) . 30
A.1 Introduction . 30
A.2 Instructions for completing the IFS proforma . 30
A.2.1 General structure of the IFS proforma . 30
A.2.2 Additional information . 30
A.3 IFS proforma . 31
A.3.1 Implementation identification . 31
A.3.2 Protocol Summary, RFC 4306 . 31
A.4 IKEv2 entities . 31
A.4.1 Roles . 31
A.4.2 IKEv2 Initiator functions . 32
A.4.2.1 IKE exchange types . 32
A.4.2.1.1 IKE SA establishment functions . 32
A.4.2.1.2 Child SA establishment functions . 32
A.4.2.1.3 INFORMATIONAL exchange functions . 33
A.4.3 IKEv2 Responder functions . 33
A.4.3.1 IKE exchange types . 33
A.4.3.1.1 IKE SA establishment functions . 33
A.4.3.1.2 Child SA establishment functions . 34
A.4.3.1.3 INFORMATIONAL exchange functions . 34
Annex B: Example IFS (TIPHON Profile of SIP, Release 3) . 35
B.1 Introduction . 35
B.2 Instructions for completing the IFS proforma . 35
B.2.1 General structure of the IFS proforma . 35
B.2.2 Additional information . 35
B.3 IFS proforma . 36
B.3.1 Implementation identification . 36
B.3.2 Protocol Summary, EN 301 xxx . 36
B.4 SIP entities . 36
B.4.1 Roles . 36
B.4.2 User Agent capabilities . 37
B.4.2.1 Registration . 37
B.4.2.2 Basic call . 37
B.4.3 Registrar capabilities . 38
B.4.3.1 Registration . 38
B.4.4 Proxy capabilities . 38
B.4.4.1 Proxy in the serving and intermediate network. 38
B.4.4.1.1 Registration . 38
B.4.4.1.2 Basic call . 38
B.4.4.2 Proxy in the home network . 39
B.4.4.2.1 Registration . 39
B.4.4.2.2 Basic call . 39
B.4.5 Gateway capabilities. 39
B.4.5.1 Basic call . 39
ETSI
5 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
Annex C: Bibliography . 40
History . 41
ETSI
6 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Methods for Testing and Specification (MTS),
and is now submitted for the ETSI standards Membership Approval Procedure.
ETSI
7 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
1 Scope
The present document, "A generic approach to interoperability testing", gives general guidance on the specification and
execution of interoperability tests for communication systems specifically in the context of product certification. It
provides a framework within which interoperability test specifications for a wide range of product types can be
developed. The guidelines are expressed as recommendations rather than strict rules and leave enough freedom to allow
test specifiers to adopt and adapt processes to suit each particular project while still ensuring that test specifications
accurately reflect the requirements of the base standards and can be executed consistently across a range of
configurations.
Interoperability testing is the structured and formal testing of functions supported remotely by two or more items of
equipment communicating by means of standardized protocols. It is not the detailed verification of protocol
requirements specified in a conformance test suite, neither is it the less formal development testing often associated
with "plug-fest" and "interop" events (frequently referred to as "bake-offs"). A methodology for the latter type of testing
is described in EG 202 810 [i.8].
Although some consideration is given within the methodology to the operating and reporting aspects of interoperability
testing, the primary focus of the present document is on the specification of interoperability testing architectures, test
plans and test suites.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
reference document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI ES 201 873-1: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; Part 1: TTCN-3 Core Language".
[i.2] Void.
[i.3] ETSI TS 101 884: "Telecommunications and Internet Protocol Harmonization Over Networks
(TIPHON) Release 3; Technology Mapping; Implementation of TIPHON architecture using SIP".
[i.4] ETSI EG 202 107: "Methods for Testing and Specification (MTS); Planning for validation and
testing in the standards-making process".
[i.5] ISO/IEC 9646 (parts 1 to 7): "Information technology - Open Systems Interconnection -
Conformance testing methodology and framework".
[i.6] IETF RFC 3261: "SIP: Session Initiation Protocol".
ETSI
8 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
[i.7] IETF RFC 4306: "Internet Key Exchange (IKEv2) Protocol".
[i.8] ETSI EG 202 810: "Methods for Testing and Specification (MTS); Automated Interoperability
Testing; Methodology and Framework".
[i.9] ETSI ES 202 553: "Methods for testing and Specification (MTS); TPLan: A notation for
expressing test Purposes".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
conformance: compliance with requirements specified in applicable standards ISO/IEC 9646 [i.5]
conformance testing: testing the extent to which an Implementation Under Test (IUT) satisfies both static and dynamic
conformance requirements ISO/IEC 9646 [i.5]
NOTE: The purpose of conformance testing is to determine to what extent a single implementation of a particular
standard conforms to the individual requirements of that standard.
device: item of software or hardware which either alone or in combination with other devices implements the
requirements of a standardized specification
Equipment Under Test (EUT): grouping of one or more devices which has not been previously shown to interoperate
with previously Qualified Equipment (QE)
Implementation Under Test (IUT): an implementation of one or more Open Systems Interconnection (OSI) protocols
in an adjacent user/provider relationship, being the part of a real open system which is to be studied by testing
(ISO/IEC 9646-1 [i.5])
interoperability: ability of two systems to interoperate using the same communication protocol
interoperability testing: activity of proving that end-to-end functionality between (at least) two communicating
systems is as required by the base standard(s) on which those systems are based
interoperability test suite: collection of test cases designed to prove the ability of two (or more) systems to
interoperate
InterWorking Function (IWF): translation of one protocol into another one so that two systems using two different
communication protocols are able to interoperate
Qualified Equipment (QE): grouping of one or more devices that has been shown and certified, by rigorous and
well-defined testing, to interoperate with other equipment
NOTE 1: Once an EUT has been successfully tested against a QE, it may be considered to be a QE, itself.
NOTE 2: Once a QE is modified, it loses its status as QE and becomes again an EUT.
System Under Test (SUT): one or more QEs and an EUT
test case: specification of the actions required to achieve a specific test purpose, starting in a stable testing state, ending
in a stable testing state and defined in either natural language for manual operation or in a machine-readable language
(such as TTCN-3) for automatic execution
test purpose: description of a well-defined objective of testing, focussing on a single interoperability requirement or a
set of related interoperability requirements
ETSI
9 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Programming Interface
EP End Point
EUT Equipment Under Test
GFT Graphical presentation Format for TTCN-3
IFS Interoperable Features Statement
IUT Implementation Under Test
IWF InterWorking Function
MMI Man-Machine Interface
MoC Means of Communication
MoT Means of Testing
OSI Open Systems Interconnection
PICS Protocol Implementation Conformance Statement
QE Qualified Equipment
SIP Session Initiation Protocol
SUT System Under Test
TP Test Purpose
TSS Test Suite Structure
4 Types of testing
Equipment implementing standardized protocols and services can be formally tested in two related but different ways,
which each have benefits and limitations:
• conformance testing can show that a product correctly implements a particular standardized protocol:
- establishes whether or not the implementation in question meets the requirements specified for the
protocol itself. For example, it will test protocol message contents and format as well as the permitted
sequences of messages.
• interoperability testing can demonstrate that a product will work with other like products:
- assesses the ability of the implementation to support the required trans-network functionality between
itself and another, similar implementation to which it is connected.
Conformance testing in conjunction with interoperability testing provides both the proof of conformance and the
guarantee of interoperation.
4.1 Interoperability testing
The term "interoperability testing" is often used in relation to the semi-formal testing carried out at multi-vendor events
as part of the product development process. While such events, often referred to as "plug-fests", "interops" and
"bake-offs", are valuable sources of information on the ability of similar products to communicate, they generally do
not offer the structured and, therefore, repeatable, testing that is an essential part of a certification scheme. For a
certification (or branding or logo) scheme to be meaningful, it is necessary that interoperability testing is carried out in
accordance with a comprehensive and structured suite of tests. In the context of the present document, it is exactly this
type of testing which is referred to as "interoperability testing". For other types of schemes, such as those arranged
between manufacturers for marketing or other purposes this approach is still valid.
NOTE: It is possible that other organizations within the global standardization community will have
interpretations of this term which differ to a greater or lesser extent.
The purpose of interoperability testing is to prove that end-to-end functionality between (at least) two communicating
systems is as required by the standard(s) on which those systems are based.
ETSI
10 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
Qualified Equipment Under
Equipment Test
Figure 1: Illustration of interoperability testing
The important factors which characterize interoperability testing are:
• the Equipment Under Test (EUT) and the Qualified Equipment (QE) together define the boundaries for testing
(figure 1);
• the EUT and QE come from different suppliers (or, at least, different product lines);
• interoperability tests are performed at interfaces that offer only normal user control and observation (i.e. not at
specialized interfaces introduced solely for testing purposes);
• interoperability tests are based on functionality as experienced by a user (i.e. they are not specified at the
protocol level). In this context a user may be human or a software application;
• the tests are performed and observed at functional interfaces such as Man-Machine Interfaces (MMIs),
protocol service interfaces and Application Programming Interfaces (APIs).
The fact that interoperability tests are performed at the end points and at functional interfaces means that
interoperability test cases can only specify functional behaviour. They cannot explicitly cause or test protocol error
behaviour.
4.2 Conformance testing
The purpose of conformance testing is to determine to what extent a single implementation of a particular standard
conforms to the individual requirements of that standard.
Conformance System Under
Test System Test
Figure 2: Illustration of conformance testing
The important factors which characterize conformance testing are as follows:
• the System or Implementation Under Test (SUT or IUT) defines the boundaries for testing (figure 2);
• the tests are executed by a dedicated test system that has full control of the SUT and the ability to observe all
communications from the SUT;
• the tests are performed at open standardized interfaces that are not (usually) accessible to a normal user.
(i.e. they are specified at the protocol level).
Because the conformance tester maintains a high degree of control over the sequence and contents of the protocol
messages sent to the IUT it is able to explore a wide range of both expected and unexpected (invalid) behaviour.
It is not within the scope of the present document to define a conformance testing methodology. However, because
interoperability testing and conformance testing complement one another, the reader of the present document would be
well-advised to study the established ISO conformance testing methodology defined in ISO/IEC 9646 parts 1 to 7 [i.5]
as applied in all ETSI conformance test specifications.
ETSI
11 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
4.3 Combining interoperability testing and conformance testing
Conformance and interoperability are both important and useful approaches to the testing of standardized protocol
implementations although it is unlikely that one will ever fully replace the other. Conformance testing is able to show
that a particular implementation complies with the protocol requirements specified in the associated base standard.
However, it is difficult for such testing to be able to prove that the implementation will interoperate with similar
implementations in other products. On the other hand, interoperability testing can clearly demonstrate that two
implementations will cooperate to provide the specified end-to-end functions but cannot easily prove that either of them
conforms to the detailed requirements of the protocol specification.
The purpose of interoperability testing is not only to show that products from different manufacturers can work together
but also to show that these products can interoperate using a specific protocol. Without this additional aspect,
interoperability testing could be considered to be almost meaningless. Within the context of standardization, it is of little
interest to know that two products can interoperate unless there is a guarantee that they are connected together by means
of a standardized protocol. It is, therefore, advisable to test the conformance of an implementation before testing for
interoperability with other (similarly tested) implementations.
Although there are quite distinct differences between conformance testing and interoperability testing, it is valid to
consider using the techniques together to give combined results. Such an approach will almost certainly involve some
compromise and it is unlikely that it would provide the breadth and depth of testing that conformance and
interoperability can offer when applied individually. However, some limited conformance testing with extensive
interoperability testing, for example, may be useful in certain situations. The test configuration shown in figure 3
permits complete interoperability testing to be undertaken while limited protocol conformance monitoring takes place.
Protocol
Monitor
QE EUT
Figure 3: Interoperability testing with conformance monitoring
While this arrangement cannot provide a complete proof of conformance, analysis of the protocol monitor output will
be able to show whether protocol signalling between the EUT and QE conformed to the appropriate standard(s)
throughout the testing.
5 Interoperability testing process overview
The present document provides users with guidelines on the main steps associated with interoperability testing. The
intention is that the guidelines should be simple and pragmatic so that the document can be used as a "cook-book" rather
than a rigid prescription of how to perform interoperability testing.
The main components of the guidelines are described in clauses 8 and 9 and are as follows:
• development of interoperability test specifications, including:
- identification of interoperable functions;
- identification of abstract architectures;
- specification of interoperability test suite structure and test purposes;
- specification of interoperability test cases.
ETSI
12 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
• the testing process, including:
- test planning;
- specification of test configurations;
- execution of the tests;
- logging results and producing test reports.
As their name implies, guidelines are only for guidance and the actual process followed should use and adapt whichever
of these guidelines are most applicable in each particular situation. In some cases this may mean the application of all
aspects.
6 Basic concepts
Figure 4 illustrates the main concepts specified in the present document. It shows the two main components of the
methodology, namely the Means of Testing (MoT) and the System Under Test (SUT). The MoT includes the roles of
test drivers and a test coordinator, the interoperability test cases and mechanisms for logging and reporting. The SUT
comprises the Equipment Under Test (EUT) and one or more items of Qualified Equipment (QE). The Means of
Communication (MoC) between the QE and the EUT is considered to be neither part of the SUT nor of the MoT.
Means of Testing (MoT)
Test Case Test Test
Interpretation Logging Reporting
Test Coordinator
System Under Test (SUT)
QE EUT
Test Interface Test Interface
Test Driver Test Driver
(QE Side) (EUT Side)
Means of communication
Figure 4: Illustration of main concepts
6.1 Means of Testing
The combination of equipment and procedures that realizes and performs the selection and execution of test cases is
known as the Means of Testing (MoT). Test cases may be executed either by a human operator or by an automated
program (see clause 6.6). The MoT should also be capable of logging test results and of producing test reports (see
clause 9.4). The MoT includes neither the System Under Test nor the means by which devices in the System Under Test
communicate.
ETSI
13 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
6.2 Equipment Under Test (EUT)
In any interoperability testing architecture there will always be one connected item which is the subject of the test. This
item is referred to as the Equipment Under Test or EUT. In general, any single test configuration will only have one
EUT (with the exception of the case illustrated in clause 6.3.2). An EUT may be end-user equipment (such as a
terminal), network equipment (such as a router) or a software application.
EUTs can be composed of any number of component parts each of which is referred to as a device. This may be a
physical device, a software package or a combination of the two. The simplest case is where the EUT is a single device.
The interconnection configuration between devices in an EUT is purely a matter for the supplier and is not prescribed in
the test architectures, nor is it considered to be an explicit part of the interoperability test for that EUT.
An EUT will not have been previously tested successfully for interoperability in a similar configuration although it may
have been tested for conformance. While this methodology does not require previous conformance testing, it is
recommended that this activity is performed, for the reasons mentioned in clause 4.3.
6.3 Qualified Equipment (QE)
6.3.1 QEs and Devices
When testing an EUT for interoperability, it is essential that the test architecture includes equipment that has already
been proven to interoperate with similar equipment from other suppliers. Such items are referred to as the Qualified
Equipment (QE). Any single test configuration may have one or more QEs. A QE may be end-user equipment (such as
a terminal), network equipment (such as a router) or a software application.
QEs can also be composed of a number of component parts, each of which is, again, referred to as a device. This may
be a physical device, a software package or a combination of the two. The simplest case is where the QE is a single
device. Thus, a QE is a collection of devices that, in a given configuration, has undergone and passed certification based
on interoperability testing.
The interconnection configuration between devices in a QE is purely a matter for the test system implementer and is not
prescribed in the test architectures.
Any given QE will have initially been tested as an EUT but, once the full range of interoperability tests have been
successfully performed, it can be considered to be a QE. However, any modification to the design or implementation of
such a QE will require it to be tested again as an EUT.
This methodology does not force an EUT to be tested against all possible QEs in the pool of QEs that may be available
in a particular testing scheme. However, the likelihood of multi-vendor interoperability is increased if it can be
demonstrated that a particular EUT interoperates with a large number of different QEs.
6.3.2 Designating the first QE
In cases of new and developing technologies, no Qualified Equipment is likely to exist. The first instance of
interoperability testing for a particular scheme will involve two (or more) EUTs rather than a number of QEs and one
EUT [i.8].
Once these EUTs are shown to successfully interoperate, they will all be designated as QEs with none having
precedence over any other. The testing scheme can then continue with new EUTs joining the pool of the existing QEs
that have already been tested in a given configuration.
It is strongly recommended that any EUT has undergone conformance testing prior to interoperability testing.
6.4 System Under Test (SUT)
The System Under Test (SUT) is the combination of one or more QEs and one single EUT (with the exception of the
case illustrated in clause 6.3.2). It does not, however, include the Means of Communication (MoC) (see clause 6.9).
ETSI
14 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
6.5 Test interface
The interfaces that are made available by the SUT in order to perform testing are known as the test interfaces. These
interfaces are accessed by the test drivers. Interfaces internal to the SUT may be used for logging and/or analysis but
they are not considered to be an essential part of the test configuration.
In the simplest case, a test interface will be the normal user interfaces offered by the product undergoing testing (EUT)
and by the QEs that are part of the SUT. Terminal equipment, for example, may be tested using a keypad, or a
point-and-click dialog, or a combination of the two. Other EUTs, such as protocol stacks, may offer an API over which
interoperability testing can be performed either manually using a terminal application or automatically using
programmable test equipment.
An EUT will offer at least one interface to either the test driver and/or the QEs.
Any interface between the SUT and the Means of Communication (see clause 6.9) is not considered to be a test
interface.
6.6 Test driver
As interoperability testing involves control and observation at the functional (rather than signalling) level,
interoperability tests should be described in terms of activities by the user of the endpoint equipment. A test driver
realizes the actions specified to be performed in a test case at one specific test interface. In many cases, this user can be
considered to be a human but in others it will be more appropriate to think of the user as an application within a
software system.
As a means of improving testing efficiency and consistency, the role of the test driver may be performed by an
automatic device programmed to carry out the specified test steps.
The following examples illustrate both of these cases:
EXAMPLE 1: Human User: A test architecture is established with two IP telephony terminals connected to the
same network supporting VoIP. Interoperability tests are specified at the terminals in terms such as
"Take telephone A off-hook; Dial the E.164 number of telephone B etc.".
EXAMPLE 2: Application User: A test architecture is established with two SIP Servers connected together but
no user terminals because at the time of testing there are no suitable applications available.
Interoperability tests are specified in terms such as "Cause INVITE message to be sent from QE to
IP address at EUT; On receipt of INVITE from QE, cause 100 TRYING message to be sent from
EUT to QE; etc.".
In the first case, the human test driver will be performing valid tasks of a normal user of the system, using only the
interfaces (e.g. MMI) offered by a product. In the second case, the test driver will be manipulating the EUT and the QE
by whatever means is possible (for example, over an API) to ensure that specific messages are sent and observed.
6.7 Test coordinator
In any given instance of testing there will be at least two interfaces over which the tests will be run (see cla
...
ETSI Guide
Methods for Testing and Specification (MTS);
Internet Protocol Testing (IPT);
Generic approach to interoperability testing
2 ETSI EG 202 237 V1.2.1 (2010-08)
Reference
REG/MTS-00129-IOP-TSTAPPROACH
Keywords
interoperability, IP, telephony, testing
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 2010.
All rights reserved.
TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 ETSI EG 202 237 V1.2.1 (2010-08)
Contents
Intellectual Property Rights . 6
Foreword . 6
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definitions and abbreviations . 8
3.1 Definitions . 8
3.2 Abbreviations . 9
4 Types of testing . 9
4.1 Interoperability testing . 9
4.2 Conformance testing. 10
4.3 Combining interoperability testing and conformance testing . 11
5 Interoperability testing process overview . 11
6 Basic concepts . 12
6.1 Means of Testing . 12
6.2 Equipment Under Test (EUT) . 13
6.3 Qualified Equipment (QE) . 13
6.3.1 QEs and Devices . 13
6.3.2 Designating the first QE . 13
6.4 System Under Test (SUT) . 13
6.5 Test interface . 14
6.6 Test driver . 14
6.7 Test coordinator . 14
6.8 Interoperability test cases . 14
6.9 Means of Communication . 15
7 Generic interoperability test architectures . 15
7.1 Test architectures with a single QE . 15
7.2 Test architectures with multiple QEs. 16
7.2.1 An example using 3 QEs . 16
7.2.2 Testing IP hosts with multiple QEs . 17
8 Developing interoperability tests . 18
8.1 Overview . 18
8.2 Specify abstract architecture. 19
8.3 Prepare draft IFS Proforma . 19
8.4 Specify Test Suite Structure . 20
8.4.1 Identify test groups . 20
8.4.2 Define test coverage within each test group . 20
8.5 Write Test Purposes. 21
8.6 Write test cases . 21
8.6.1 Pre-test conditions . 21
8.6.2 Test steps and verdicts . 22
8.6.2.1 Test steps . 22
8.6.2.2 Verdicts . 22
8.6.2.3 Specification of test steps and verdicts. 22
8.6.3 Example . 23
8.6.4 Pre-amble and post-amble . 24
8.6.4.1 Alternative test case presentation forms . 25
8.7 Validate test cases . 26
8.8 Finalize IFS . 26
9 Interoperability testing process . 26
ETSI
4 ETSI EG 202 237 V1.2.1 (2010-08)
9.1 Overview . 26
9.2 Prepare for testing . 27
9.2.1 Test arrangement . 27
9.2.2 Test planning . 28
9.3 Testing . 29
9.3.1 Manual testing . 29
9.3.2 Automated testing . 29
9.4 Write test report . 29
Annex A: Example IFS (Internet Key Exchange protocol, IKEv2) . 30
A.1 Introduction . 30
A.2 Instructions for completing the IFS proforma . 30
A.2.1 General structure of the IFS proforma . 30
A.2.2 Additional information . 30
A.3 IFS proforma . 31
A.3.1 Implementation identification . 31
A.3.2 Protocol Summary, RFC 4306 . 31
A.4 IKEv2 entities . 31
A.4.1 Roles . 31
A.4.2 IKEv2 Initiator functions . 32
A.4.2.1 IKE exchange types . 32
A.4.2.1.1 IKE SA establishment functions . 32
A.4.2.1.2 Child SA establishment functions . 32
A.4.2.1.3 INFORMATIONAL exchange functions . 33
A.4.3 IKEv2 Responder functions . 33
A.4.3.1 IKE exchange types . 33
A.4.3.1.1 IKE SA establishment functions . 33
A.4.3.1.2 Child SA establishment functions . 34
A.4.3.1.3 INFORMATIONAL exchange functions . 34
Annex B: Example IFS (TIPHON Profile of SIP, Release 3) . 35
B.1 Introduction . 35
B.2 Instructions for completing the IFS proforma . 35
B.2.1 General structure of the IFS proforma . 35
B.2.2 Additional information . 35
B.3 IFS proforma . 36
B.3.1 Implementation identification . 36
B.3.2 Protocol Summary, EN 301 xxx . 36
B.4 SIP entities . 36
B.4.1 Roles . 36
B.4.2 User Agent capabilities . 37
B.4.2.1 Registration . 37
B.4.2.2 Basic call . 37
B.4.3 Registrar capabilities . 38
B.4.3.1 Registration . 38
B.4.4 Proxy capabilities . 38
B.4.4.1 Proxy in the serving and intermediate network. 38
B.4.4.1.1 Registration . 38
B.4.4.1.2 Basic call . 38
B.4.4.2 Proxy in the home network . 39
B.4.4.2.1 Registration . 39
B.4.4.2.2 Basic call . 39
B.4.5 Gateway capabilities. 39
B.4.5.1 Basic call . 39
ETSI
5 ETSI EG 202 237 V1.2.1 (2010-08)
Annex C: Bibliography . 40
History . 41
ETSI
6 ETSI EG 202 237 V1.2.1 (2010-08)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Methods for Testing and Specification (MTS).
ETSI
7 ETSI EG 202 237 V1.2.1 (2010-08)
1 Scope
The present document, "A generic approach to interoperability testing", gives general guidance on the specification and
execution of interoperability tests for communication systems specifically in the context of product certification. It
provides a framework within which interoperability test specifications for a wide range of product types can be
developed. The guidelines are expressed as recommendations rather than strict rules and leave enough freedom to allow
test specifiers to adopt and adapt processes to suit each particular project while still ensuring that test specifications
accurately reflect the requirements of the base standards and can be executed consistently across a range of
configurations.
Interoperability testing is the structured and formal testing of functions supported remotely by two or more items of
equipment communicating by means of standardized protocols. It is not the detailed verification of protocol
requirements specified in a conformance test suite, neither is it the less formal development testing often associated
with "plug-fest" and "interop" events (frequently referred to as "bake-offs"). A methodology for the latter type of testing
is described in EG 202 810 [i.8].
Although some consideration is given within the methodology to the operating and reporting aspects of interoperability
testing, the primary focus of the present document is on the specification of interoperability testing architectures, test
plans and test suites.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
reference document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI ES 201 873-1: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; Part 1: TTCN-3 Core Language".
[i.2] Void.
[i.3] ETSI TS 101 884: "Telecommunications and Internet Protocol Harmonization Over Networks
(TIPHON) Release 3; Technology Mapping; Implementation of TIPHON architecture using SIP".
[i.4] ETSI EG 202 107: "Methods for Testing and Specification (MTS); Planning for validation and
testing in the standards-making process".
[i.5] ISO/IEC 9646 (parts 1 to 7): "Information technology - Open Systems Interconnection -
Conformance testing methodology and framework".
[i.6] IETF RFC 3261: "SIP: Session Initiation Protocol".
ETSI
8 ETSI EG 202 237 V1.2.1 (2010-08)
[i.7] IETF RFC 4306: "Internet Key Exchange (IKEv2) Protocol".
[i.8] ETSI EG 202 810: "Methods for Testing and Specification (MTS); Automated Interoperability
Testing; Methodology and Framework".
[i.9] ETSI ES 202 553: "Methods for testing and Specification (MTS); TPLan: A notation for
expressing test Purposes".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
conformance: compliance with requirements specified in applicable standards ISO/IEC 9646 [i.5]
conformance testing: testing the extent to which an Implementation Under Test (IUT) satisfies both static and dynamic
conformance requirements ISO/IEC 9646 [i.5]
NOTE: The purpose of conformance testing is to determine to what extent a single implementation of a particular
standard conforms to the individual requirements of that standard.
device: item of software or hardware which either alone or in combination with other devices implements the
requirements of a standardized specification
Equipment Under Test (EUT): grouping of one or more devices which has not been previously shown to interoperate
with previously Qualified Equipment (QE)
Implementation Under Test (IUT): an implementation of one or more Open Systems Interconnection (OSI) protocols
in an adjacent user/provider relationship, being the part of a real open system which is to be studied by testing
(ISO/IEC 9646-1 [i.5])
interoperability: ability of two systems to interoperate using the same communication protocol
interoperability testing: activity of proving that end-to-end functionality between (at least) two communicating
systems is as required by the base standard(s) on which those systems are based
interoperability test suite: collection of test cases designed to prove the ability of two (or more) systems to
interoperate
InterWorking Function (IWF): translation of one protocol into another one so that two systems using two different
communication protocols are able to interoperate
Qualified Equipment (QE): grouping of one or more devices that has been shown and certified, by rigorous and
well-defined testing, to interoperate with other equipment
NOTE 1: Once an EUT has been successfully tested against a QE, it may be considered to be a QE, itself.
NOTE 2: Once a QE is modified, it loses its status as QE and becomes again an EUT.
System Under Test (SUT): one or more QEs and an EUT
test case: specification of the actions required to achieve a specific test purpose, starting in a stable testing state, ending
in a stable testing state and defined in either natural language for manual operation or in a machine-readable language
(such as TTCN-3) for automatic execution
test purpose: description of a well-defined objective of testing, focussing on a single interoperability requirement or a
set of related interoperability requirements
ETSI
9 ETSI EG 202 237 V1.2.1 (2010-08)
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Programming Interface
EP End Point
EUT Equipment Under Test
GFT Graphical presentation Format for TTCN-3
IFS Interoperable Features Statement
IUT Implementation Under Test
IWF InterWorking Function
MMI Man-Machine Interface
MoC Means of Communication
MoT Means of Testing
OSI Open Systems Interconnection
PICS Protocol Implementation Conformance Statement
QE Qualified Equipment
SIP Session Initiation Protocol
SUT System Under Test
TP Test Purpose
TSS Test Suite Structure
4 Types of testing
Equipment implementing standardized protocols and services can be formally tested in two related but different ways,
which each have benefits and limitations:
• conformance testing can show that a product correctly implements a particular standardized protocol:
- establishes whether or not the implementation in question meets the requirements specified for the
protocol itself. For example, it will test protocol message contents and format as well as the permitted
sequences of messages.
• interoperability testing can demonstrate that a product will work with other like products:
- assesses the ability of the implementation to support the required trans-network functionality between
itself and another, similar implementation to which it is connected.
Conformance testing in conjunction with interoperability testing provides both the proof of conformance and the
guarantee of interoperation.
4.1 Interoperability testing
The term "interoperability testing" is often used in relation to the semi-formal testing carried out at multi-vendor events
as part of the product development process. While such events, often referred to as "plug-fests", "interops" and
"bake-offs", are valuable sources of information on the ability of similar products to communicate, they generally do
not offer the structured and, therefore, repeatable, testing that is an essential part of a certification scheme. For a
certification (or branding or logo) scheme to be meaningful, it is necessary that interoperability testing is carried out in
accordance with a comprehensive and structured suite of tests. In the context of the present document, it is exactly this
type of testing which is referred to as "interoperability testing". For other types of schemes, such as those arranged
between manufacturers for marketing or other purposes this approach is still valid.
NOTE: It is possible that other organizations within the global standardization community will have
interpretations of this term which differ to a greater or lesser extent.
The purpose of interoperability testing is to prove that end-to-end functionality between (at least) two communicating
systems is as required by the standard(s) on which those systems are based.
ETSI
10 ETSI EG 202 237 V1.2.1 (2010-08)
Qualified Equipment Under
Equipment Test
Figure 1: Illustration of interoperability testing
The important factors which characterize interoperability testing are:
• the Equipment Under Test (EUT) and the Qualified Equipment (QE) together define the boundaries for testing
(figure 1);
• the EUT and QE come from different suppliers (or, at least, different product lines);
• interoperability tests are performed at interfaces that offer only normal user control and observation (i.e. not at
specialized interfaces introduced solely for testing purposes);
• interoperability tests are based on functionality as experienced by a user (i.e. they are not specified at the
protocol level). In this context a user may be human or a software application;
• the tests are performed and observed at functional interfaces such as Man-Machine Interfaces (MMIs),
protocol service interfaces and Application Programming Interfaces (APIs).
The fact that interoperability tests are performed at the end points and at functional interfaces means that
interoperability test cases can only specify functional behaviour. They cannot explicitly cause or test protocol error
behaviour.
4.2 Conformance testing
The purpose of conformance testing is to determine to what extent a single implementation of a particular standard
conforms to the individual requirements of that standard.
Conformance System Under
Test System Test
Figure 2: Illustration of conformance testing
The important factors which characterize conformance testing are as follows:
• the System or Implementation Under Test (SUT or IUT) defines the boundaries for testing (figure 2);
• the tests are executed by a dedicated test system that has full control of the SUT and the ability to observe all
communications from the SUT;
• the tests are performed at open standardized interfaces that are not (usually) accessible to a normal user.
(i.e. they are specified at the protocol level).
Because the conformance tester maintains a high degree of control over the sequence and contents of the protocol
messages sent to the IUT it is able to explore a wide range of both expected and unexpected (invalid) behaviour.
It is not within the scope of the present document to define a conformance testing methodology. However, because
interoperability testing and conformance testing complement one another, the reader of the present document would be
well-advised to study the established ISO conformance testing methodology defined in ISO/IEC 9646 parts 1 to 7 [i.5]
as applied in all ETSI conformance test specifications.
ETSI
11 ETSI EG 202 237 V1.2.1 (2010-08)
4.3 Combining interoperability testing and conformance testing
Conformance and interoperability are both important and useful approaches to the testing of standardized protocol
implementations although it is unlikely that one will ever fully replace the other. Conformance testing is able to show
that a particular implementation complies with the protocol requirements specified in the associated base standard.
However, it is difficult for such testing to be able to prove that the implementation will interoperate with similar
implementations in other products. On the other hand, interoperability testing can clearly demonstrate that two
implementations will cooperate to provide the specified end-to-end functions but cannot easily prove that either of them
conforms to the detailed requirements of the protocol specification.
The purpose of interoperability testing is not only to show that products from different manufacturers can work together
but also to show that these products can interoperate using a specific protocol. Without this additional aspect,
interoperability testing could be considered to be almost meaningless. Within the context of standardization, it is of little
interest to know that two products can interoperate unless there is a guarantee that they are connected together by means
of a standardized protocol. It is, therefore, advisable to test the conformance of an implementation before testing for
interoperability with other (similarly tested) implementations.
Although there are quite distinct differences between conformance testing and interoperability testing, it is valid to
consider using the techniques together to give combined results. Such an approach will almost certainly involve some
compromise and it is unlikely that it would provide the breadth and depth of testing that conformance and
interoperability can offer when applied individually. However, some limited conformance testing with extensive
interoperability testing, for example, may be useful in certain situations. The test configuration shown in figure 3
permits complete interoperability testing to be undertaken while limited protocol conformance monitoring takes place.
Protocol
Monitor
QE EUT
Figure 3: Interoperability testing with conformance monitoring
While this arrangement cannot provide a complete proof of conformance, analysis of the protocol monitor output will
be able to show whether protocol signalling between the EUT and QE conformed to the appropriate standard(s)
throughout the testing.
5 Interoperability testing process overview
The present document provides users with guidelines on the main steps associated with interoperability testing. The
intention is that the guidelines should be simple and pragmatic so that the document can be used as a "cook-book" rather
than a rigid prescription of how to perform interoperability testing.
The main components of the guidelines are described in clauses 8 and 9 and are as follows:
• development of interoperability test specifications, including:
- identification of interoperable functions;
- identification of abstract architectures;
- specification of interoperability test suite structure and test purposes;
- specification of interoperability test cases.
ETSI
12 ETSI EG 202 237 V1.2.1 (2010-08)
• the testing process, including:
- test planning;
- specification of test configurations;
- execution of the tests;
- logging results and producing test reports.
As their name implies, guidelines are only for guidance and the actual process followed should use and adapt whichever
of these guidelines are most applicable in each particular situation. In some cases this may mean the application of all
aspects.
6 Basic concepts
Figure 4 illustrates the main concepts specified in the present document. It shows the two main components of the
methodology, namely the Means of Testing (MoT) and the System Under Test (SUT). The MoT includes the roles of
test drivers and a test coordinator, the interoperability test cases and mechanisms for logging and reporting. The SUT
comprises the Equipment Under Test (EUT) and one or more items of Qualified Equipment (QE). The Means of
Communication (MoC) between the QE and the EUT is considered to be neither part of the SUT nor of the MoT.
Means of Testing (MoT)
Test Case Test Test
Interpretation Logging Reporting
Test Coordinator
System Under Test (SUT)
QE EUT
Test Interface Test Interface
Test Driver Test Driver
(QE Side) (EUT Side)
Means of communication
Figure 4: Illustration of main concepts
6.1 Means of Testing
The combination of equipment and procedures that realizes and performs the selection and execution of test cases is
known as the Means of Testing (MoT). Test cases may be executed either by a human operator or by an automated
program (see clause 6.6). The MoT should also be capable of logging test results and of producing test reports (see
clause 9.4). The MoT includes neither the System Under Test nor the means by which devices in the System Under Test
communicate.
ETSI
13 ETSI EG 202 237 V1.2.1 (2010-08)
6.2 Equipment Under Test (EUT)
In any interoperability testing architecture there will always be one connected item which is the subject of the test. This
item is referred to as the Equipment Under Test or EUT. In general, any single test configuration will only have one
EUT (with the exception of the case illustrated in clause 6.3.2). An EUT may be end-user equipment (such as a
terminal), network equipment (such as a router) or a software application.
EUTs can be composed of any number of component parts each of which is referred to as a device. This may be a
physical device, a software package or a combination of the two. The simplest case is where the EUT is a single device.
The interconnection configuration between devices in an EUT is purely a matter for the supplier and is not prescribed in
the test architectures, nor is it considered to be an explicit part of the interoperability test for that EUT.
An EUT will not have been previously tested successfully for interoperability in a similar configuration although it may
have been tested for conformance. While this methodology does not require previous conformance testing, it is
recommended that this activity is performed, for the reasons mentioned in clause 4.3.
6.3 Qualified Equipment (QE)
6.3.1 QEs and Devices
When testing an EUT for interoperability, it is essential that the test architecture includes equipment that has already
been proven to interoperate with similar equipment from other suppliers. Such items are referred to as the Qualified
Equipment (QE). Any single test configuration may have one or more QEs. A QE may be end-user equipment (such as
a terminal), network equipment (such as a router) or a software application.
QEs can also be composed of a number of component parts, each of which is, again, referred to as a device. This may
be a physical device, a software package or a combination of the two. The simplest case is where the QE is a single
device. Thus, a QE is a collection of devices that, in a given configuration, has undergone and passed certification based
on interoperability testing.
The interconnection configuration between devices in a QE is purely a matter for the test system implementer and is not
prescribed in the test architectures.
Any given QE will have initially been tested as an EUT but, once the full range of interoperability tests have been
successfully performed, it can be considered to be a QE. However, any modification to the design or implementation of
such a QE will require it to be tested again as an EUT.
This methodology does not force an EUT to be tested against all possible QEs in the pool of QEs that may be available
in a particular testing scheme. However, the likelihood of multi-vendor interoperability is increased if it can be
demonstrated that a particular EUT interoperates with a large number of different QEs.
6.3.2 Designating the first QE
In cases of new and developing technologies, no Qualified Equipment is likely to exist. The first instance of
interoperability testing for a particular scheme will involve two (or more) EUTs rather than a number of QEs and one
EUT [i.8].
Once these EUTs are shown to successfully interoperate, they will all be designated as QEs with none having
precedence over any other. The testing scheme can then continue with new EUTs joining the pool of the existing QEs
that have already been tested in a given configuration.
It is strongly recommended that any EUT has undergone conformance testing prior to interoperability testing.
6.4 System Under Test (SUT)
The System Under Test (SUT) is the combination of one or more QEs and one single EUT (with the exception of the
case illustrated in clause 6.3.2). It does not, however, include the Means of Communication (MoC) (see clause 6.9).
ETSI
14 ETSI EG 202 237 V1.2.1 (2010-08)
6.5 Test interface
The interfaces that are made available by the SUT in order to perform testing are known as the test interfaces. These
interfaces are accessed by the test drivers. Interfaces internal to the SUT may be used for logging and/or analysis but
they are not considered to be an essential part of the test configuration.
In the simplest case, a test interface will be the normal user interfaces offered by the product undergoing testing (EUT)
and by the QEs that are part of the SUT. Terminal equipment, for example, may be tested using a keypad, or a
point-and-click dialog, or a combination of the two. Other EUTs, such as protocol stacks, may offer an API over which
interoperability testing can be performed either manually using a terminal application or automatically using
programmable test equipment.
An EUT will offer at least one interface to either the test driver and/or the QEs.
Any interface between the SUT and the Means of Communication (see clause 6.9) is not considered to be a test
interface.
6.6 Test driver
As interoperability testing involves control and observation at the functional (rather than signalling) level,
interoperability tests should be described in terms of activities by the user of the endpoint equipment. A test driver
realizes the actions specified to be performed in a test case at one specific test interface. In many cases, this user can be
considered to be a human but in others it will be more appropriate to think of the user as an application within a
software system.
As a means of improving testing efficiency and consistency, the role of the test driver may be performed by an
automatic device programmed to carry out the specified test steps.
The following examples illustrate both of these cases:
EXAMPLE 1: Human User: A test architecture is established with two IP telephony terminals connected to the
same network supporting VoIP. Interoperability tests are specified at the terminals in terms such as
"Take telephone A off-hook; Dial the E.164 number of telephone B etc.".
EXAMPLE 2: Application User: A test architecture is established with two SIP Servers connected together but
no user terminals because at the time of testing there are no suitable applications available.
Interoperability tests are specified in terms such as "Cause INVITE message to be sent from QE to
IP address at EUT; On receipt of INVITE from QE, cause 100 TRYING message to be sent from
EUT to QE; etc.".
In the first case, the human test driver will be performing valid tasks of a normal user of the system, using only the
interfaces (e.g. MMI) offered by a product. In the second case, the test driver will be manipulating the EUT and the QE
by whatever means is possible (for example, over an API) to ensure that specific messages are sent and observed.
6.7 Test coordinator
In any given instance of testing there will be at least two interfaces over which the tests will be run (see clause 6.5). The
test coordinator is responsible for managing the two (or more) test drivers and synchronizing their actions, if needed.
The test coordinator is only a conceptual role and, in a practical case of testing, this role may be taken b
...
SLOVENSKI STANDARD
01-oktober-2016
0HWRGH]DSUHVNXãDQMHLQVSHFLILFLUDQMH 076 3UHVNXãDQMHLQWHUQHWQHJD
SURWRNROD*HQHULþQLSULVWRSNSUHVNXãDQMXPHGREUDWRYDOQRVWL
Methods for Testing and Specification (MTS) - Internet Protocol Testing (IPT) - Generic
approach to interoperability testing
Ta slovenski standard je istoveten z: ETSI EG 202 237 V1.2.1 (2010-08)
ICS:
33.040.01 Telekomunikacijski sistemi Telecommunication systems
na splošno in general
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.
Loading comments...