ETSI ETS 300 406 ed.1 (1995-04)

Methods for Testing and Specification (MTS); Protocol and profile conformance testing specifications; Standardization methodology

DE/MTS-012

Metode za preskušanje in specificiranje (MTS) – Specifikacije za preskušanje skladnosti protokola in profila – Metodologija standardiziranja

General Information

• Status: Published
• Publication Date: 20-Apr-1995
• Technical Committee: MTS - Methods for Testing & Specification
• Current Stage: 12 - Completion
• Due Date: 17-Apr-1995
• Completion Date: 21-Apr-1995
• Ref Project: SIST ETS 300 406 E1:2003 - Methods for Testing and Specification (MTS); Protocol and profile conformance testing specifications; Standardization methodology

Standards Content (Sample)

SLOVENSKI STANDARD
SIST ETS 300 406 E1:2003
01-december-2003
Metode za preskušanje in specificiranje (MTS) – Specifikacije za preskušanje
skladnosti protokola in profila – Metodologija standardiziranja
Methods for Testing and Specification (MTS); Protocol and profile conformance testing
specifications; Standardization methodology
Ta slovenski standard je istoveten z: ETS 300 406 Edition 1
ICS:
33.020 Telekomunikacije na splošno Telecommunications in
general
SIST ETS 300 406 E1:2003 en
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------

SIST ETS 300 406 E1:2003

---------------------- Page: 2 ----------------------

SIST ETS 300 406 E1:2003
EUROPEAN ETS 300 406
TELECOMMUNICATION April 1995
STANDARD
Source: ETSI TC-MTS Reference: DE/MTS-00012
ICS: 33.020, 33.040.40
Key words: Testing, methodology
Methods for Testing and Specification (MTS);
Protocol and profile conformance testing specifications;
Standardization methodology
ETSI
European Telecommunications Standards Institute
ETSI Secretariat
Postal address: F-06921 Sophia Antipolis CEDEX - FRANCE
Office address: 650 Route des Lucioles - Sophia Antipolis - Valbonne - FRANCE
X.400: c=fr, a=atlas, p=etsi, s=secretariat - Internet: [e-mail]
Tel.: [phone] - Fax: [phone]
Copyright Notification: No part may be reproduced except as authorized by written permission. The copyright and the
foregoing restriction extend to reproduction in all media.
© European Telecommunications Standards Institute 1995. All rights reserved.

---------------------- Page: 3 ----------------------

SIST ETS 300 406 E1:2003
Page 2
ETS 300 406: April 1995
Whilst every care has been taken in the preparation and publication of this document, errors in content,
typographical or otherwise, may occur. If you have comments concerning its accuracy, please write to
"ETSI Editing and Committee Support Dept." at the address shown on the title page.

---------------------- Page: 4 ----------------------

SIST ETS 300 406 E1:2003
Page 3
ETS 300 406: April 1995
Contents
Foreword.7
Introduction .7
1 Scope .9
2 Normative references . 10
3 Definitions and abbreviations. 11
3.1 Definitions. 11
3.2 Terms which can cause confusion. 13
3.3 Abbreviations. 13
4 Introduction to conformance testing . 15
4.1 What is conformance testing? . 15
4.1.1 Context and history of conformance testing . 15
4.1.2 Scope of conformance testing . 15
4.2 Conforming implementation . 16
4.2.1 Conformance requirement. 16
4.2.2 Conformance statement. 16
4.3 Technical objectives of conformance testing. 17
4.4 The need for standardized test specifications. 17
5 Conformance testing applied to base specification or to profile . 18
5.1 What is a base specification?. 18
5.2 What is a profile?. 19
5.3 Examples of profiles. 20
5.3.1 In EWOS, CEN/CENELEC. 20
5.3.2 In ETSI. 21
5.4 Structure of a profile specification. 21
5.5 Conformance to a profile . 22
5.6 Conformance testing applied to base specification. 22
5.7 Conformance testing applied to profile. 23
5.8 Decision criteria: base specification of profile testing?. 23
6 The PICS and other base specification ICS; a companion document for base specifications. 26
6.1 Objective of the PICS (base specification ICS). 26
6.2 What is a PICS proforma (base specification ICS proforma) . 27
6.3 PICS (ICS) proforma and TS . 27
6.4 Semantics of PICS (ICS) proforma status notations. 28
6.5 PICS proforma (base specification ICS proforma) criteria . 30
6.5.1 ISO/IEC 9646 criteria. 30
6.5.2 Additional criteria. 30
6.6 Standardization of a PICS (ICS) proforma. 30
7 TS for a base specification. 31
7.1 The production process . 32
7.2 Synopsis of components and their dependencies. 35
7.3 Checklist of components. 38
7.4 Definitions of the components and criteria . 39
7.4.1 TSS&TP. 39
7.4.1.1 TSS. 39
7.4.1.1.1 General rules and recommendations . 39

---------------------- Page: 5 ----------------------

SIST ETS 300 406 E1:2003
Page 4
ETS 300 406: April 1995
7.4.1.1.2 Recommendation concerning the testing of "state
transitions" .40
7.4.1.1.3 Recommendation concerning the "inopportune" groups .41
7.4.1.1.4 Recommendation and rule concerning the "variation"
groups .41
7.4.1.1.5 Rules and recommendations concerning the naming
conventions .42
7.4.1.2 TPs .44
7.4.1.3 CTPs .46
7.4.1.3.1 Why combine TPs?.46
7.4.1.3.2 How to combine TPs.47
7.4.1.3.3 TSS for CTPs. 50
7.4.2 ATM. 51
7.4.3 ATS conventions.51
7.4.4 ATS.52
7.4.5 TCP .52
7.4.6 Partial PIXIT proforma (partial IXIT proforma) .53
7.4.7 PCTR proforma.53
7.4.8 Untestable TPs.53
7.4.9 Abstract selection rules.54
7.4.10 ATS to TP map . 54
7.5 Amendments. 54
7.5.1 TSS & TP amendment .54
7.5.2 ATS amendment.55
8 Profile ICS proforma.55
8.1 Objective of the profile ICS .55
8.2 What is a profile ICS proforma? .56
8.2.1 Profile requirement list profile RL .57
8.2.2 Profile-specific ICS proforma.58
8.3 Profile ICS proforma and TS.58
8.4 Profile ICS and conformance assessment process.58
8.4.1 Profile ICS and static conformance review.58
8.4.2 Profile ICS and test case parameterization, de-selection .58
8.4.3 Profile ICS components attached to test reports.59
9 TS for a profile.59
9.1 Principles of profile testing .59
9.2 A word on the production process .61
9.3 Synopsis of components and their dependencies .62
9.4 Checklist of components .63
9.5 Definitions of the components and criteria .64
9.5.1 Profile-specific test specification.64
9.5.2 PTS-summary .66
9.5.3 SCTR proforma tailored to the profile.66
9.5.4 Profile IXIT XRL. 66
9.5.5 Profile IXIT proforma .66
10 Standardization of test specifications .67
10.1 Physical contents of a conformance TS.67
10.1.1 Structure of a conformance TS.67
10.1.1.1 Conformance TS for a base specification. 67
10.1.1.2 Conformance TS for a profile. 70
10.1.2 Need for an overview document.71
10.1.3 Editing rules for TTCN.71
10.1.4 Copyright of proformas .72
10.1.5 Copyright of TTCN.MP. 73
10.1.6 Conformance clause .73
10.1.7 Case of ATSs having parts in common.73
10.2 Minimal quality criteria .74

---------------------- Page: 6 ----------------------

SIST ETS 300 406 E1:2003
Page 5
ETS 300 406: April 1995
10.2.1 Reference to the reference specification . 75
10.2.2 Components of a TS. 75
10.2.3 TTCN notation. 75
10.2.4 Validation. 75
10.2.4.1 Validation of TSS&TP . 75
10.2.4.2 Validation of ATS specification. 75
11 Conformance clause. 76
11.1 Conformance with this ETS. 76
11.2 Conformance with ISO/IEC 9646. 76
Annex A (normative): Normative criteria for the standardization of PICS (ICS) proforma
specifications. 78
A.1 Static versus dynamic requirements in an ICS proforma . 78
A.2 Level of detail of an ICS proforma . 79
A.3 Requirements on numeric values. 81
A.4 References to ICS items. 83
A.5 Scope of qualified optionals. 84
Annex B (informative): Components of a TS for a base specification when several ATSs are
produced. 86
B.1 Why a single ATM has been chosen . 86
B.2 Overview .87
B.3 Synopsis of the components and their dependencies. 87
B.4 Relationships between components . 89
B.5 Additional TSS&TP. 89
B.6 Consequences on the pre-existing ATS. 89
Annex C (informative): Guidance on the combination of TPs. 90
C.1 Tests having the same morphology. 90
C.2 Tests having the same spirit. 92
Annex D (informative): Guidance on the definition of a test strategy. 93
D.1 Test strategy: initial steps . 93
D.1.1 Why do the protocols have to be tested? . 93
D.1.2 What will an implementation to test look like?. 93
D.2 Test strategy: choice of test method. 94
D.2.1 One or several test method(s)?. 95
D.2.2 Technical criteria intrinsic to standardized area. 95
D.2.3 Technical criteria related to implementation conditions. 97
D.2.4 Other criteria . 98
D.2.5 Further clarification on test methods characteristics. 98
D.2.6 Clarification concerning the concept of upper tester . 98
D.3 Test strategy: coverage and costs. 99

---------------------- Page: 7 ----------------------

SIST ETS 300 406 E1:2003
Page 6
ETS 300 406: April 1995
D.4 Test strategy: technical refinements. 100
Annex E (informative): Bibliography. 101
History . 102

---------------------- Page: 8 ----------------------

SIST ETS 300 406 E1:2003
Page 7
ETS 300 406: April 1995
Foreword
This European Telecommunication Standard (ETS) has been produced by the Methods for Testing and
Specification (MTS) Technical Committee of the European Telecommunications Standards Institute (ETSI).
This ETS provides rules and guidance on how to develop telecommunications conformance testing
specifications with the final objective to standardize them.
ETSI has explored innovative solutions for the long term, and at present, a "proposer" in harmonized
approaches to the pressing requirement of developing testing standards in various areas of
telecommunications. It is also essential that ETSI remains aware of what is going on in the information
technology area outside ETSI, in close relations with its own technical activities.
The production of conformance testing standards in ETSI has already started in various technical areas,
and needs to be continued with the best harmonization of efforts.
ETSI TC-MTS recommends that the TC and STC chairmen in their role of co-ordinators of standard
development ensure that the groups of experts working in conformance testing in their bodies are fully
aware of the status of affairs in this area so that initiatives are co-ordinated properly. Special care should
be devoted to the consistency of the methodology used (test specifications' structure, test notations,
criteria for acceptance, etc.), to the need of not duplicating efforts and to the strategy of improving
convergence for testing at the European and international levels.
With the co-operation of all ETSI members, an initial orientation towards a harmonized approach to
conformance testing standards and practice is desirable and possible. This is why the TC MTS work
program includes the production of a comprehensive methodology guide that will represent a further step
forward in harmonization and consistency.
Proposed transposition dates
Date of latest announcement of this ETS (doa): 31 July 1995
Date of latest publication of new National Standard
or endorsement of this ETS (dop/e): 31 January 1996
Date of withdrawal of any conflicting National Standard (dow): 31 January 1996
Introduction
This ETS is one element of a series of documents produced during 1992 and 1993 by ETSI. These ETSs
will provide a reference library of rules and guidance on how to standardize telecommunications
conformance testing specifications in Europe. These rules and guidance take into account the state of the
art, the international and European standards and reports already published on that matter, the specifics of
test suite editing, and the practices of the ETSI and CEN secretariats for the production of European
standards.
Although this ETS has been produced in order to become a reference document for ETSI, it is designed to
be used in a wider European context, for example, as a technical basis for accreditation and specification
purposes, as well as for European Commission initiated conformance testing programmes, e.g. the
Conformance Testing Services (CTS) programme.

---------------------- Page: 9 ----------------------

SIST ETS 300 406 E1:2003
Page 8
ETS 300 406: April 1995
Blank page

---------------------- Page: 10 ----------------------

SIST ETS 300 406 E1:2003
Page 9
ETS 300 406: April 1995
1 Scope
This European Telecommunication Standard (ETS) specifies a methodology for developing
telecommunications conformance testing specifications with the final objective to standardize them.
This ETS defines the components to be produced when specifying a European conformance Testing
Standard (TS) in the domain of telecommunications.
The methodology specified in this ETS is, in principle, applicable to domains other than
telecommunications, e.g. Open Systems Interconnection (OSI). However, the limits of such an application
of the methodology outside telecommunications testing are not defined, and each group defining test
specifications are responsible for deciding if any part of this methodology is applicable.
This methodology applies to conformance TSs for protocols, for profiles, for information objects, for
interfaces, and for services.
NOTE 1: The methodology of this ETS is generic, in the sense that it applies to protocols, to
profiles, to information objects, etc. Following the example of ISO/IEC 9646
(references [5] to [12]), the terminology employed is also generic, e.g. Implementation
Conformance Statement (ICS), Implementation eXtra Information for Testing (IXIT).
However, in order to avoid confusing the readers who are familiar with the protocol-
related terminology, the non-generic terms (Protocol Implementation Conformance
Statement (PICS), Protocol Implementation eXtra Information for Testing (PIXIT), etc.)
have been kept in the titles of clauses and subclauses.
The test of physical aspects, such as "layer 1", analogue measurements, Electro-Magnetic Compatibility
(EMC), is not covered by this ETS. However, part of the methodology may be still be applied, e.g. the
concepts of ICS, IXIT, Test Purposes (TPs), and Test Suite Structure (TSS).
This ETS provides the rules for standardization bodies developing telecommunications conformance TSs.
NOTE 2: One objective of this ETS is the use, when relevant, within standardization bodies. For
this purpose, within ETSI, it has the same value as an internal Technical Committee
Reference Technical Report (TCR-TR).
This ETS is applicable to organizations such as Conformance Testing Services (CTS) programmes, pre-
standardization institutions, private institutions, which develop conformance testing specifications with the
objective of contributing them for the purposes of European telecommunications standardization.
This ETS contains two types of information, following a twofold objective:
- it clarifies the principles of the ISO/IEC 9646 (references [5] to [12]) testing framework and
methodology, as well as the concept of profile;
- it defines additional criteria for European telecommunications TSs, whereby it constitutes a
"European telecommunications profile" of the ISO/IEC 9646 (references [5] to [12]) methodology,
containing its own recommendations.
This ETS is consistent with the normative references listed in clause 2. This ETS extends the current
principles contained in ISO/IEC 9646 (references [5] to [12]), in particular in the field of the combination of
TPs, and concerning the choice of a single Abstract Test Suite (ATS).
For conformance testing in the "voluntary domain" (i.e. outside the regulatory domain), for instance by the
suppliers to support their commercial claims for conformance, TSs are normally published as I-ETS, or
ETS, depending on their role and their maturity.
This ETS addresses telecommunications TS methodology for the voluntary domain.

---------------------- Page: 11 ----------------------

SIST ETS 300 406 E1:2003
Page 10
ETS 300 406: April 1995
2 Normative references
This ETS incorporates by dated and undated reference, provisions from other publications. These
normative references are cited at the appropriate places in the text and the publications are listed
hereafter. For dated references, subsequent amendments to or revisions of any of these publications apply
to this ETS only when incorporated in it by amendment or revision. For undated references the latest
edition of the publication referred to applies.
[1] ETR 153: "Methods for Testing and Specification (MTS); Guidance on the
production and completion of System Conformance Test Report (SCTR) and
Protocol Conformance Test Report (PCTR) proformas".
[2] ETR 141: "Methods for Testing and Specification (MTS); Protocol and profile
conformance testing specifications; The Tree and Tabular Combined Notation
(TTCN) style guide".
[3] PNE-Rules (1991): "CEN/CENELEC Internal Regulations - IR; Part 3: Rules for
the drafting and presentation of European Standards (PNE-Rules)".
[4] EN 45001 (1989): "General criteria for the operation of testing laboratories".
[5] ISO/IEC 9646-1 Edition 2 (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
1: General concepts" (including DAM1 and DAM2, to be published).
[6] ISO/IEC 9646-2 Edition 2 (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework -
Part 2: Abstract Test Suite Specification" (including DAM1 and DAM2, to be
published).
[7] ISO/IEC 9646-3: "Information technology - Open Systems Interconnection -
Conformance testing methodology and framework - Part 3: The Tree and
Tabular Combined Notation".
[8] ISO/IEC 9646-3 AM 1: "Information technology - Open Systems Interconnection
- Conformance testing methodology and framework - Part 3: The Tree and
Tabular Combined Notation. Amendment 1: TTCN Extensions (concurrent
TTCN)".
NOTE 1: ISO/IEC 9646-3 AM 1 will be included in the next edition of ISO/IEC 9646-3, which will
contain TTCN and concurrent TTCN in the same standard.
[9] ISO/IEC 9646-4 IS Edition 2: (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
4: Test Realisation" (including DAM1 and DAM2, to be published).
[10] ISO/IEC 9646-5 IS Edition 2: (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
5: Requirements on test laboratories and clients for the Conformance
Assessment Process" (including DAM1 and DAM2, to be published).
[11] ISO/IEC 9646-6 IS (1994): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework -
Part 6: Protocol Profile Test Specification".
[12] ISO/IEC 9646-7 DIS (1993): "Information technology - Open Systems
Interconnection - Conformance testing methodology and framework - Part
7: Implementation Conformance Statements".

---------------------- Page: 12 ----------------------

SIST ETS 300 406 E1:2003
Page 11
ETS 300 406: April 1995
NOTE 2: The clauses of ISO/IEC 9646 (references [5] to [12]) referenced in this ETS are from
edition 2: 1994 (unpublished). Although the contents of ISO/IEC 9646 have changed, it
is useful to know the references to edition 1: 1991. The correspondence is as follows:
Edition 1: 1991 Edition 2: 1994
Part 2, clause 12 Part 2, clause 11
Part 1, annex B Part 2, annex A
3 Definitions and abbreviations
3.1 Definitions
For the purposes of this ETS, all the definitions in ISO/IEC 9646 (references [5] to [12]) apply.
In addition, the following definitions apply, or have been re-formulated for clarification:
Abstract Test Method (ATM) (Abstract Testing Method): Definition of ISO/IEC 9646-1 [5], subclause
3.3.5, applies.
NOTE 1: The ISO definition, complemented with the explanation of ISO/IEC 9646-2 [6],
subclause 11.1.7, properly specifies that the ATM contains "enough detail to enable
Abstract Test Cases (ATCs) to be specified for this test method". This means that the
ATM is a detailed description of the testing architecture. Conversely, the four methods
for Single Party Testing (SPy
...