ETSI EG 202 237 V1.2.1 (2010-08)

Final draft ETSI EG 202 237 V1.2.1 (2010-06)
ETSI Guide
Methods for Testing and Specification (MTS);
Internet Protocol Testing (IPT);
Generic approach to interoperability testing
---------------------- Page: 1 ----------------------
2 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
Reference
REG/MTS-00129-IOP-TstApproach
Keywords
interoperability, IP, telephony, testing
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org

The present document may be made available in more than one electronic version or in print. In any case of existing or

perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).

In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive

Users of the present document should be aware that the document may be subject to revision or change of status.

Information on the current status of this and other ETSI documents is available at

If you find errors in the present document, please send your comment to one of the following services:

DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered

3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.

GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.

Intellectual Property Rights ................................................................................................................................ 6

Foreword ............................................................................................................................................................. 6

1 Scope ........................................................................................................................................................ 7

2 References ................................................................................................................................................ 7

2.1 Normative references ......................................................................................................................................... 7

2.2 Informative references ........................................................................................................................................ 7

3 Definitions and abbreviations ................................................................................................................... 8

3.1 Definitions .......................................................................................................................................................... 8

3.2 Abbreviations ..................................................................................................................................................... 9

4 Types of testing ........................................................................................................................................ 9

4.1 Interoperability testing ....................................................................................................................................... 9

4.2 Conformance testing......................................................................................................................................... 10

4.3 Combining interoperability testing and conformance testing ........................................................................... 11

5 Interoperability testing process overview ............................................................................................... 11

6 Basic concepts ........................................................................................................................................ 12

6.1 Means of Testing .............................................................................................................................................. 12

6.2 Equipment Under Test (EUT) .......................................................................................................................... 13

6.3 Qualified Equipment (QE) ............................................................................................................................... 13

6.3.1 QEs and Devices ......................................................................................................................................... 13

6.3.2 Designating the first QE ............................................................................................................................. 13

6.4 System Under Test (SUT) ................................................................................................................................ 13

6.5 Test interface .................................................................................................................................................... 14

6.6 Test driver ........................................................................................................................................................ 14

6.7 Test coordinator ................................................................................................................................................ 14

6.8 Interoperability test cases ................................................................................................................................. 14

6.9 Means of Communication ................................................................................................................................ 15

7 Generic interoperability test architectures .............................................................................................. 15

7.1 Test architectures with a single QE .................................................................................................................. 15

7.2 Test architectures with multiple QEs................................................................................................................ 16

7.2.1 An example using 3 QEs ............................................................................................................................ 16

7.2.2 Testing IP hosts with multiple QEs ............................................................................................................ 17

8 Developing interoperability tests ............................................................................................................ 18

8.1 Overview .......................................................................................................................................................... 18

8.2 Specify abstract architecture............................................................................................................................. 19

8.3 Prepare draft IFS Proforma .............................................................................................................................. 19

8.4 Specify Test Suite Structure ............................................................................................................................. 20

8.4.1 Identify test groups ..................................................................................................................................... 20

8.4.2 Define test coverage within each test group ............................................................................................... 20

8.5 Write Test Purposes.......................................................................................................................................... 21

8.6 Write test cases ................................................................................................................................................. 21

8.6.1 Pre-test conditions ...................................................................................................................................... 21

8.6.2 Test steps and verdicts ................................................................................................................................ 22

8.6.2.1 Test steps ............................................................................................................................................... 22

8.6.2.2 Verdicts ................................................................................................................................................. 22

8.6.2.3 Specification of test steps and verdicts.................................................................................................. 22

8.6.3 Example ...................................................................................................................................................... 23

8.6.4 Pre-amble and post-amble .......................................................................................................................... 24

8.6.4.1 Alternative test case presentation forms ................................................................................................ 25

8.7 Validate test cases ............................................................................................................................................ 26

8.8 Finalize IFS ...................................................................................................................................................... 26

9 Interoperability testing process .............................................................................................................. 26

9.1 Overview .......................................................................................................................................................... 26

9.2 Prepare for testing ............................................................................................................................................ 27

9.2.1 Test arrangement ........................................................................................................................................ 27

9.2.2 Test planning .............................................................................................................................................. 28

9.3 Testing .............................................................................................................................................................. 29

9.3.1 Manual testing ............................................................................................................................................ 29

9.3.2 Automated testing ....................................................................................................................................... 29

9.4 Write test report ................................................................................................................................................ 29

Annex A: Example IFS (Internet Key Exchange protocol, IKEv2) ................................................... 30

A.1 Introduction ............................................................................................................................................ 30

A.2 Instructions for completing the IFS proforma ........................................................................................ 30

A.2.1 General structure of the IFS proforma .............................................................................................................. 30

A.2.2 Additional information ..................................................................................................................................... 30

A.3 IFS proforma .......................................................................................................................................... 31

A.3.1 Implementation identification .......................................................................................................................... 31

A.3.2 Protocol Summary, RFC 4306 ......................................................................................................................... 31

A.4 IKEv2 entities ......................................................................................................................................... 31

A.4.1 Roles ................................................................................................................................................................. 31

A.4.2 IKEv2 Initiator functions .................................................................................................................................. 32

A.4.2.1 IKE exchange types .................................................................................................................................... 32

A.4.2.1.1 IKE SA establishment functions ........................................................................................................... 32

A.4.2.1.2 Child SA establishment functions ......................................................................................................... 32

A.4.2.1.3 INFORMATIONAL exchange functions .............................................................................................. 33

A.4.3 IKEv2 Responder functions ............................................................................................................................. 33

A.4.3.1 IKE exchange types .................................................................................................................................... 33

A.4.3.1.1 IKE SA establishment functions ........................................................................................................... 33

A.4.3.1.2 Child SA establishment functions ......................................................................................................... 34

A.4.3.1.3 INFORMATIONAL exchange functions .............................................................................................. 34

Annex B: Example IFS (TIPHON Profile of SIP, Release 3) ............................................................. 35

B.1 Introduction ............................................................................................................................................ 35

B.2 Instructions for completing the IFS proforma ........................................................................................ 35

B.2.1 General structure of the IFS proforma .............................................................................................................. 35

B.2.2 Additional information ..................................................................................................................................... 35

B.3 IFS proforma .......................................................................................................................................... 36

B.3.1 Implementation identification .......................................................................................................................... 36

B.3.2 Protocol Summary, EN 301 xxx ....................................................................................................................... 36

B.4 SIP entities .............................................................................................................................................. 36

B.4.1 Roles ................................................................................................................................................................. 36

B.4.2 User Agent capabilities .................................................................................................................................... 37

B.4.2.1 Registration ................................................................................................................................................. 37

B.4.2.2 Basic call ..................................................................................................................................................... 37

B.4.3 Registrar capabilities ........................................................................................................................................ 38

B.4.3.1 Registration ................................................................................................................................................. 38

B.4.4 Proxy capabilities ............................................................................................................................................. 38

B.4.4.1 Proxy in the serving and intermediate network........................................................................................... 38

B.4.4.1.1 Registration ........................................................................................................................................... 38

B.4.4.1.2 Basic call ............................................................................................................................................... 38

B.4.4.2 Proxy in the home network ......................................................................................................................... 39

B.4.4.2.1 Registration ........................................................................................................................................... 39

B.4.4.2.2 Basic call ............................................................................................................................................... 39

B.4.5 Gateway capabilities......................................................................................................................................... 39

B.4.5.1 Basic call ..................................................................................................................................................... 39

Annex C: Bibliography .......................................................................................................................... 40

History .............................................................................................................................................................. 41

IPRs essential or potentially essential to the present document may have been declared to ETSI. The information

pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found

in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in

respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee

can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web

This ETSI Guide (EG) has been produced by ETSI Technical Committee Methods for Testing and Specification (MTS),

The present document, "A generic approach to interoperability testing", gives general guidance on the specification and

execution of interoperability tests for communication systems specifically in the context of product certification. It

provides a framework within which interoperability test specifications for a wide range of product types can be

developed. The guidelines are expressed as recommendations rather than strict rules and leave enough freedom to allow

test specifiers to adopt and adapt processes to suit each particular project while still ensuring that test specifications

accurately reflect the requirements of the base standards and can be executed consistently across a range of

Interoperability testing is the structured and formal testing of functions supported remotely by two or more items of

equipment communicating by means of standardized protocols. It is not the detailed verification of protocol

requirements specified in a conformance test suite, neither is it the less formal development testing often associated

with "plug-fest" and "interop" events (frequently referred to as "bake-offs"). A methodology for the latter type of testing

Although some consideration is given within the methodology to the operating and reporting aspects of interoperability

testing, the primary focus of the present document is on the specification of interoperability testing architectures, test

References are either specific (identified by date of publication and/or edition number or version number) or

non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the

Referenced documents which are not found to be publicly available in the expected location might be found at

NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee

The following referenced documents are necessary for the application of the present document.

The following referenced documents are not necessary for the application of the present document but they assist the

[i.1] ETSI ES 201 873-1: "Methods for Testing and Specification (MTS); The Testing and Test Control

[i.3] ETSI TS 101 884: "Telecommunications and Internet Protocol Harmonization Over Networks

(TIPHON) Release 3; Technology Mapping; Implementation of TIPHON architecture using SIP".

[i.4] ETSI EG 202 107: "Methods for Testing and Specification (MTS); Planning for validation and

[i.5] ISO/IEC 9646 (parts 1 to 7): "Information technology - Open Systems Interconnection -

[i.8] ETSI EG 202 810: "Methods for Testing and Specification (MTS); Automated Interoperability

[i.9] ETSI ES 202 553: "Methods for testing and Specification (MTS); TPLan: A notation for

For the purposes of the present document, the following terms and definitions apply:

conformance: compliance with requirements specified in applicable standards ISO/IEC 9646 [i.5]

conformance testing: testing the extent to which an Implementation Under Test (IUT) satisfies both static and dynamic

NOTE: The purpose of conformance testing is to determine to what extent a single implementation of a particular

device: item of software or hardware which either alone or in combination with other devices implements the

Equipment Under Test (EUT): grouping of one or more devices which has not been previously shown to interoperate

Implementation Under Test (IUT): an implementation of one or more Open Systems Interconnection (OSI) protocols

in an adjacent user/provider relationship, being the part of a real open system which is to be studied by testing

interoperability: ability of two systems to interoperate using the same communication protocol

interoperability testing: activity of proving that end-to-end functionality between (at least) two communicating

interoperability test suite: collection of test cases designed to prove the ability of two (or more) systems to

InterWorking Function (IWF): translation of one protocol into another one so that two systems using two different

Qualified Equipment (QE): grouping of one or more devices that has been shown and certified, by rigorous and

NOTE 1: Once an EUT has been successfully tested against a QE, it may be considered to be a QE, itself.

NOTE 2: Once a QE is modified, it loses its status as QE and becomes again an EUT.

test case: specification of the actions required to achieve a specific test purpose, starting in a stable testing state, ending

in a stable testing state and defined in either natural language for manual operation or in a machine-readable language

test purpose: description of a well-defined objective of testing, focussing on a single interoperability requirement or a

Equipment implementing standardized protocols and services can be formally tested in two related but different ways,

• conformance testing can show that a product correctly implements a particular standardized protocol:

- establishes whether or not the implementation in question meets the requirements specified for the

protocol itself. For example, it will test protocol message contents and format as well as the permitted

• interoperability testing can demonstrate that a product will work with other like products:

- assesses the ability of the implementation to support the required trans-network functionality between

Conformance testing in conjunction with interoperability testing provides both the proof of conformance and the

The term "interoperability testing" is often used in relation to the semi-formal testing carried out at multi-vendor events

as part of the product development process. While such events, often referred to as "plug-fests", "interops" and

"bake-offs", are valuable sources of information on the ability of similar products to communicate, they generally do

not offer the structured and, therefore, repeatable, testing that is an essential part of a certification scheme. For a

certification (or branding or logo) scheme to be meaningful, it is necessary that interoperability testing is carried out in

accordance with a comprehensive and structured suite of tests. In the context of the present document, it is exactly this

type of testing which is referred to as "interoperability testing". For other types of schemes, such as those arranged

between manufacturers for marketing or other purposes this approach is still valid.

NOTE: It is possible that other organizations within the global standardization community will have

The purpose of interoperability testing is to prove that end-to-end functionality between (at least) two communicating